natedredd10

Actually though, if you are ok with flushing to binary files weekly, have you considered not even bothering with a db? Its hard to understand what you would be gaining from a db really with that time frame, unless these are baby steps of a much larger idea. If you search on elitetrader for "tick database" or "tick db" and go back a few years there are some interesting discussions...In retrospect those discussions boiled down to morons like me trying to figure out how to use HDF5, berkeley db...monetdb now although I think thats too new to have come up on elite a few years ago. Then there are guys in those discussions who realized this was a waste of time and just went with flat binary files...Don't even want to think about how much analysis they have done vs the time I've spent on this stuff... Maybe I'm just hard headed but pytables/HDF5 is my last stand then I'm just going with binary files until its a problem... this discussion will give you all the leads to search on you want in this area: Nuclear Phynance

Hack the market blog on HDF5 is about the only good info ive found on tick db construction: Hack the market billions and billions Hack the market managing tick data with hdf5 Hack the market tick data & hdf5 (part 2) From what I've found the biggest thing is how many instruments you want to be logging. If you only want to store a few then go with one of the open source relational packages but keep in mind it probably wouldn't be to hard to max out performance with a non time series db if you start adding instruments down the line. Trying to roll my own tick db from parts has been a really demoralizing experience to be honest. Its a pretty thin number of users so there isn't so much to go on. Retail is using commercial solutions from the charting software and then institutions are using ultra expensive time series solutions like KDB+..so you are really on your own being in the middle.

There is no sense in going out of your way to learn OOP... You don't lose money from admitting that you can't stand the OOP paradigm personally....OOP is for computer scientist, not for the "hackers" like basically anyone reading this. This is a great announcement from the standpoint of the computer scientists it will lure into giving stuff away to hackers and the easy to use functions that will eventually become standard easy language stuff for "dummies"... The only reason to jump on this is if you want to sell stuff easylang wise. Neoticker is an overpriced POS that to me seems to have been written in visual basic ...great concepts behind the project but it ran out of steam years ago, it was too ambitious for its time. If there was any juice left in the project it would have been rewrote 2 years ago. Ninja is the opposite of neoticker...most ultra modern retail software, trying so desperately to smash the 2005 version of tradestation, but they have opened up too many can of worms to not get koed with the next TS release... I went with IB over tradestation when I started out just because the TWS demo made me feel safer than the nothing from TS. TS management is clearly brilliant in letting the competition hang themselves, learn from then then go in for the kill.. I watched a little CNBC at my parents today for the first time since the crisis and noticed a thinkorswim commercial...They have black boxed trading functions like in modular audio software(reaktor, Sync,Kyma) that you dont need to know a language, just concepts and then connect wires. I didn't see a tradestation commercial....I remember seeing TS commercials on CNBC when I started trading. Ditmar, if you really have managements ear you should try to convince them to make a better demo...I'm sure I'm not the only one who is not going to basically escrow tradestation $5K to demo their platform. Maybe that made sense with limited alternatives, but that was half a decade ago. I'm almost certain they would have got a piece of my business if they had any rational sense of a "demo version" of the software...Putting 5k in escrow for a software demo to me would only make sense with like 50mil to swing % wise, just to demo a product..Its just totally unreasonable and why Ninja is not out of business, yet.

Anyone with speculative common sense can pick up on the stench in this article, thats why I believe Brownsfan walks the walk but is the only candle trader in the world I would actually bet on but gets offended when his method is put down. The alarm should have went off when this comp sci/HFT guy spends more on news than on his other data..huh? "800k to 1.5 million shares a day and make 2cents per trade on average. He made 7 figures profit in 2008 after expenses and taxes." Lets forget "expenses"(whatever that is sopposed to mean) and taxes and just use 1 mil gross... He has to make 50 trades to make a dollar with an expectancy of 2 cents per trade. To make 7 figures min he has to make $4k a day($1 mil / 250 trading days) ... $4000 / .02 cents means he is making 200,000 trades a day.I would have a very hard time believing that if he was even talking SPY. Yet this magician makes 200k trades a day martingaling market neutral strategies but goes out of his way to point out he is not making a market...implies he is an option trader yet trades "shares"...complete fucking nonsense. Ironically though, I basically agree with what he said, I could have wrote that myself. I don't mind the skill saw to the brain though ego wise when intial experiments fail and guys like brownsfan make way more than my zero...this guy is making shit up. I would put him on a failed options trader that got Hull from nuclearphy and found faitail.org(been there done that)..then made up this nonsense to feel good about himself.

I think there should be an end to "technical analysis" because its a completely poisoned term at this point. It's just an old term, chosen to sound scientific back in the day..Just like the term "quant" has evolved because it sounds "science fiction" like. Most very wealthy people are not self made but born into wealth. "quant" is a great marketting strategy to get to play with these peoples money because at the mean they are of average intelligence...it was daddy or grandpa who were the above average ones. We are all dealing with numbers and statistics..no one trading is using a data feed that contains something other than numbers. In any form of analysis there is science and there is a nonsense. Just because you are not crunching numbers does not mean you are not using science. People of average intelligence are easily fooled by nonsense. Its like if I was a great discretionary MA based trader I would never start a CTA under the category of discretionary TA based. MY CTA would be based on a strategy of a type of finite impulse response filter, optimized by a biologicaly based non linear pattern search..... Message boards and open discussion have been poisoned by this fake wall...TA vs quant..TA vs fundy.. people actually did science before the advent of the computer...

Here are applications I've found that are market related, but nothing in the way of theory/concepts: SVM on RSI Support Vector Machine RSI System Quantum Financier Decision tree bagging system on GLD Max Dama on Automated Trading: Decision Tree Bagging System (R code) this guy has a bunch of different algos on market data, some with code. Intelligent Trading

The problem to me with retail software in this area is its using some strange assumptions as far as feature extraction. Its easy to argue that for ES, price is a feature, volume is a feature, cash index price is a feature of, the volume of GS is a feature of ES. I don't see how you can make an argument though that the RSI, MA, slow stochastic of ES price is a feature...Its making a huge assumption there is predictive power in these things to start with and that they are something other than simple price summaries. To me the whole interesting thing in this area is in ditching these old models of price prediction and going straight to the factors that we know matter and creating new models.

That book I posted above is by the far the easiest to follow I've found, its kind of a machine learning for dummies book which may be why MseTrap thinks its crap. I don't have the background to gain much of anything from the most highly recommended books in the field and I do think you have to be open to the idea that if you don't have the background this may be a huge waste of time. I also have Bioinformatics: The Machine Learning Approach by Pierre Baldi..Maybe the only thing I find easier about it though is that I find bioinformatics interesting in and of itself so the concepts are not so abstract as with a straight ML textbook. I think without the background its important to find text that give real world examples on real datasets that interest you since there is no book on ML with financial datasets at its base that I know of. A straight up ML textbook is as much fun to read as your highschool calc book.

This book is the only one I've found that I'm starting to actually make some sense of NN. Amazon.com: Machine Learning: An Algorithmic Perspective (Chapman & Hall/Crc Machine Learning & Pattern Recognition)… Unfortunately, It doesn't get as deep into other machine learning algorithms as it does NN. Ranger, you should check out hidden markov models. I don't know the math to make heads or tails of them but Rentech hired the Baum in this algo awhile back: http://en.wikipedia.org/wiki/Baum-Welch_algorithm As far as other books there are a lot of machine learning bioinformatics books that I think are easier to follow than straight ML textbooks. Also has the nice feature that the problem domain is a bit similar as far as large data sets and not totally sure what it is you are looking for.

The biggest problem with this type of stuff is in setting the thresholds for what tells the computer to not view things as a zone forming. Like in that above example at the start on the upper part of the zone, that one bar violates the zone by quite a lot and what is telling you to ignore that comes because of the forward bars after that. If you look at the attached picture, programming the box on the left to capture the zone would be easy but would never really pick up what you want unless you line it up in the past. What you probably need is something like on the right, with smaller windows based off a few number of bars that set a larger window, that sets a larger window, that sets a larger window until you get what you want. Much much harder to do since you also have to keep track of something like if you picture 3 iterations zone forming 1 = true zone forming 2 = maybe zone forming 3 = maybe zone forming 4 = maybe zone formed = maybe zone forming 1 = true zone forming 2 = true zone forming 3 = maybe zone forming 4 = maybe zone formed = maybe zone forming 1 = true zone forming 2 = true zone forming 3 = FALSE zone forming 4 = null zone formed = null False can't just scrap everything and start over, although maybe you could have it backtrack one level to maybe again and keep going.

On the other hand its not hard to imagine systems that could not be traded/analyzed manually. Retail algo trading has never really progressed beyond the line of thinking you would use with a 166mhz pentium, 1 gig of hd space and and a data feed hooked up to a single instrument and this is reflected in our retail software. All its really doing with modern cpu cycles is the same type of 1985 analysis faster. Things surely are not helped that practically all the new automated trading ideas of the last 20 years fall under "quant" if your looking for books and there is a level of sophistication that quickly becomes uneconomic to produce retail software for.

Beware of the man who tries to sell you an ATM machine at a discount...They will of course always give you a "deal" on such machinery..... To me on this chart you are making a rookie mistake that every stock chart is either a long or short... Most stocks, if randomly picked from a basket are "wait and see" in reality and you have to sit in the weeds like a marine sniper for your target knowing what you are shooting at. I wouldn't do anything here other than wait and see if the chart bull flags, especially if people getting scared risk wise at the index.

buy low sell high... I dropped out of being a gold bug in 2007ish but I'm shocked here how low gold is really... Greece/Euro stuff isn't even sending it through the roof after a global financial crisis as a baseline floor... You have to be nuts to not error on the side of the false breakout here given the fundys.

abstract...hmm...20 characters...

haha, well first I love those kind of jumbled word things you posted...I've seen those before and it really bothers me that I feel I can read those much much faster than actual english, probably because the brain starts to skim for the releventant information and actually becomes optimal vs reading every whole word. A counter though is I doubt you could read a whole book like that. Like trying to prove a sprinter is a better marathon runner than a marathon runner by measuring the first 100 meters. I do agree on the "context" part of this discussion. CONTEXT: the parts of a piece of writing, speech, etc., that precede and follow a word or passage and contribute to its full meaning. Its simply lazy though to say that a computer can not do this...to me its like saying the context of something is not quantifiable because its based on "magic". Discretionary trading wise I consider myself a tape reader, I have no idea though how to write an "indicator" of what I'm doing. Its certainly not though because what I'm doing is using "magic", its only because I don't currently know how to quanitify what I'm doing. If you look up what a poisson distribution is used to make predictions with probabilistically, if you had never heard of a poisson distribution it sounds like "magic". IMO market "context" is that the brain can pick up patterns in the markets unstable distribution better than those grafting stable distribution time series analysis ideas on to an unstable distribution. Thats why an MA cross system can work for a discretionary trader IF they know the context to pull the trigger and why you are BSing yourself if you try to walk forward a 2 parameter MA cross system automated wise. The automated system isn't just ignoring the single variable of "context", its ignoring a shitload of variables because context is a shitload of variables. Its ammusing for some reason with trading people like to turn to the logic of "OR" as to why most people can't do this. Assuming everything is correct in the system besides one part of it when they start running good and making assumptions on the last part they tinkered with as being the solution... Pyschology turned the corner, risk management turned the corner, started making things more simple tactically, added 1 indicator, subtracted 1 indicator, changed time frames to longer, changed time frames to shorter, changed from candlesticks to whatever, blah blah. You never hear of a winning trader who moved to renko charts or whatever and started losing money. As if "winning" traders NEVER tinker around with what they do to try to make more cash... If you don't start from the logic of AND, that everything you have read and know is wrong and you have to prove it correct...Starting from nothing, I'm wrong tacticaly AND on risk management AND pyschology AND...ect...you get lost on the forest. Retail traders are a lazy bunch in general, selection bias would dicate this game will attract a lot of people looking to make "easy" money who simply do not like to "work". Not to mention you have to have some degree of Narcissistic Personality Disorder to believe you are smarter than almost everyone on the globe and willing to play a high barrier to entry gambling game on it. When you cut the fat away, 99% retail fails because they are lazy narcissist AND can not only take the pounding/variance in the pocket book, but the mirror to their ego that they are not the smartest person on earth when that pocket book takes any kind of hit. The reason to stay in the retail game if you have that degree of narcissim and pocket book is "smart money" is argueably more stupid actually at this point, but just has deeper pockets to withstand variance and company structure makes it easy to pass the blame, so its a highly exploitable situation for those who are not lazy at the retail level. If you disagree its highly likely you have never met the actual "boogie men" who run "smart money" and what complete dipshits they are. Not talking of course the overlords of the hedge fund industry, but the big bank "whales".

This may have more to do with the topic of the lecture and who it would attract rather than matlab not being used in trading. If you look on nuclearphy or willmot its pretty ubiquitous. I doubt anyone here would have to deal with vector sizes that true high frequency guys would be into. this blog gives some idea what it can do for trading Trading With MATLAB If you look at how small the code is to produce a market profile its rather impressive. My experience with it was through getting my cousin to get me a student license. The syntax of the language just doesn't "jive" with my brain for whatever reason. Just like VBA is sopposed to be much easier than C# but VBA makes no sense to me at all while C# i thought was the easiest language to learn syntax wise. For backtesting mathworks has this webinar Algorithmic Trading with MATLAB for Financial Applications - Recorded Webinar Matlab's strengths vs retail trading software is in data visualization(just look at the 3d sharpe ration plot of the backtest of a simple MA cross system in that webinar), hooking into basically any database you could want already is there as matlab functions and any kind of statistical function you could want can be had for a price if not free. There is a Dickey–Fuller test for ninja around and just compare the code to matlab or R. A jumbled mess that who even knows if the algorithm is correct in ninja, you have to put your faith in the 1 guy that wrote it. Matlab/R is not going to have wrong statistical algorithms in it exactly because its not specifically for trading. Not to mention the speed you get from piggy backing the work of actual computer scientist, statisticians and mathematicians across fields of interest using the same tool vs some single generous retail trader giving away their algorithm. Not a hard bet that they are not writing the algorithm at its optimal speed, even assuming its output is correct(a HUGE if). I soppose this all depends on context of the original question. I have a very low opinion of the ability to take OHLC data, data snoop a bunch of windowed "indicators" and do anything beyond simple variance walk forward wise with any meaningful sample of trades. No offense to Investors R/T, which I think is a lovely charting package but algo wise...these are irrational expectations.

No..you are very fortunate as far as an initial mindset goes with this line of thinking. You are going to shoot yourself learning there is no way around it, all you have to do is not shoot yourself in the head as you learn and end the game. You also just have to stay in the game long enough for your self inflicted wounds to heal and not give in to the "pain". You will either learn or you won't but at least you will give yourself a shot. Stops below/above a swing low/high is about as good as you can expect right now. Being put on tilt because of a single loss is something that takes practice and gets less and less the more you are put on tilt. You can read all the psychology nonsense you want but the only way to train your brain really is to lose money many many times when you think you are correct. Almost any kind of gambling that you can gain an edge in is better than markets because of the cost..micro sports betting account, micro poker account.. My niece is due in the fall...As soon as she has a concept of money I'm going to start training her to be a trader. Put a small amount of glue on the tail side of a coin to make it unfair, she is always heads...Give her 200 bucks in a piggy bank that she can only bet vs me in this unfair coin game in her favor and drive her nuts with random streaks betting 10 cents a flip...

I've come to the point that 2 - 24 inch monitors and my old 19 inch mounted vertically is all I will let myself have. At some point displaying more data is actually going to make trade decisions more difficult. I figure if I feel like I need more space than this then I need to figure out a better way to interface with the data on the software side. I picked up 2 second hand white boards off craigs list that I would highly recommend. There is just something about getting away from the computer to sketch out ideas on such a large surface, that can't be saved and isn't actually dealing with real data but just concepts that feels more contemplative. If you really want to be cool you can draw a bunch of charts and equations on the white board and load up the most wacky charts on your screens when you have a woman over...Can easily inflate your pnl by 10 to her and its simply impossible for her to question it... I've never done that...of course......

Well this is kind of what I'm working on. You should look into bayesian analysis ideas and the concept of priors. The computer does what its good at as far as crunching data but the initial parameters are set and variable based on my speculative talent. Its hardly non trivial though, after a year I still don't have anything thats better than just looking at a chart. I think you also have to keep in mind that the computer is good at looking at the vast interrelationships between markets and index components that would be impossible to chart and do analysis on yourself. I'm not really sure you can beat a well trained brain on straight out time series analysis for a single market given the problem though. Manually marking off zones certainly cuts down on the variables but the variables are still massive for the computer to handle. I also don't think you can really do this with candle sticks and not tick data. The more trades the system puts on backtesting the more random the results are from the information that is being tossed out with an ohlc summary of the tick data. this is a pretty cool blog post from today though that i would think would be a piece of the puzzle for what you are trying to do. http://intelligenttradingtech.blogspot.com/2010/06/quantitative-candlestick-pattern.html

I respect the forex robot guys for their brilliant marketing. Calling an automated system a "robot" and then selling it to fools is something I'm pissed I didn't think of.

meh, I can see why people with vast resources would take matlab all day but at least for me the price and productivity just are not even close. The TradeAnalytics package is still being worked on but should be pretty sweet when its done. https://r-forge.r-project.org/R/?group_id=316

I think this is ultimately the million dollar question, literally, once you have systems that have positive expectation. Of course if we knew with 100% certainty that a trade had positive expectation, even only making 1 penny after transaction cost there is no rational argument that can be made for not taking every positive EV bet that comes your way. That of course would be foolish in the real world as 1 penny would obviously not compensate for our uncertainty of the model/system's expectation. This to me is the heart of speculation and a good place to come at this stuff. How much do I need to get paid to bet on the certainty of the expectation of this model/system? Something else to think about is that obviously over leveraging is a death blow to any system but on the other hand the farther you slide away from geometric growth from winning trades, the more you increase you risk of ruin by making the draw down padding less from large winning trades.

As usual the market evolves past what the literature and networked trading board information can keep up with between us. In COT terms, the idea you can separate in real time/short time frame/day trader time frame the difference between commercial and large speculator action is completely absurd unless you ignore the entire concept of index arbitrage. Small speculators mean nothing to the market, they basically only add noise to the analysis because of their lack of liquidity. Its not as if "commercial" index futures players are so underfunded that they have not already blasted back against high frequency "speculative" index arbs. That doesn't even make any sense, of course they have. If anything they are better funded. Ditmar's trade intensity I would imagine is measuring when the spread between the tick by tick value of the cash index gets enough out of whack with the derivative that all the arbs start gun slinging...some pull out a piece of alpha, a shitload react to that but are too late and on the tape you tend to get an easy fade of that game. It however also draws a line in the sand for the discretionary trader on a short time frame of what algo arbs are looking at as far as the spread and "fair value" for the current implied index volatility at that moment, but that info is completely out of context if you are only looking at price levels of the derivative. I would imagine though its information that adds to positive expectation when added to other trading decisions, even though fundamentally its obviously suboptimal in theory. You could improve on this with a ms resolution snapshot of the actual value of the cash index and compare things, but taking ms snapshots of the cash index at our level is hardly non trivial.

Pat, I like to pull your chain and try to be a bigger ass on trading boards than the perception people have of you, its amusing to me to do so since there is largely so little signal to pull of trading boards in general. I'm sure from our exchanges there is no way you realize that I absolutely want to be you in 20 years. Of course I can't DEMONSTRATE comparable work, yet...I put you on being between 52-56 years of age.. so I have 2 decades to beat what you have done. I noticed you didn't respond on 2+2 to the guy that called you out for throwing out jargon in that thread you created in business and finance. Your mistake is that you don't think anyone else at the trading board level is thinking as deeply about this stuff as you. I could care less about your data feed. I meant literally what DB to learn here to store as much tick data as possible. Why SQL over a time series db? I'm on the fence between HDF5 and Berkley..I can understand if you have hired some DB guy who is simply good with SQL...any analysis of the situation though makes it obvious that SQL is not optimal for tick data at all. However, obviously a poorly done time series db setup by an amateur may be suboptimal to an SQL guru in reality. As far as swarm intelligence, I don't see how you get around that functions that are modified by their input are better than a sum cascade of functions modeling input in the long run if you are really talking about "intelligence". In the short term obviously yes...I'm hardly though trying to figure out this most difficult problem tomorrow. That swarm intelligence book is pretty lame though really after you have read it. I would think you would find The Ants by Bert Holldobler much more interesting. Its the most lovely book I've ever purchased to boot.

I've spent the last 6 months with a student version of Matlab trying to learn it and have finally given up and just going with R. Matlab seems to update too frequently that the online examples of certain things are just a huge cluster fuck. You end up spending more time debugging examples than learning anything concept wise. R just always seems to work, and if something is broken it points you to how to fix it. Python has some cool stuff but I don't see why you would bother learning Python over R for this game unless you are already a very good Python programmer. Even if you are a good programmer in a more traditional language, the value of something like R is that the algorithms have already been gone over with a fine tooth comb for you. Probably 10 guys who know the language better than you and 10 guys who are vastly better at algorithm analysis than you have already added their 2 cents for free. If you find something then you can just spend your time debugging the translation of the algorithm itself to a more efficient execution strategy, not that R doesn't have some nice parallel processing features though for brute force analysis. Good luck starting from nothing with a C variant.. As far as retail backtesting software, its all utter garbage. Its not rational to expect it to ever get better than garbage if aimed at retail trading given the failure rate. The focus will always be on the lowest common denominator in order to stay in business.