darthtrader2.0

The markit ABX indexs have been interesting to watch through this whole thing. This last selloff looks like the AAA paper finally was hammered nicely more than the junk paper. Paul Van Eeden is my fav macro guy, he thinks the best long term bet here is to short long bonds and bet that the yield curve has to steepen to clear the system. http://howestreet.com/audiovideo/index.php?pl=/fbn/index.php/mediaplayer/284 Equity wise it just seems like there is just enough bad news to send us to the bottom of the range but what can possibly happen now to break that range? Same thing at the top in reverse. So any of you guys swing trading the index long here with your longer term money? Buy and hold doesn't make much sense to me but damn..its hard to ask for a more defined range to trade mid term than this.

The bayesian part of my brain I've been cultivating can't possibly disagree with that statement more...down with frequentism and its illusory "facts"!

I just got "An Introduction to High-Frequency Finance"..I don't see anyway around wading into the waters of High-Frequency if you want to be truely empirical about order flow. Financial Markets Tick By Tick by Pierre Lequeux looks like another cool book on this subject. The problem I already see though with this stuff is our tools are absolutely not designed for real High-Frequency analysis. It would be simplistic to overwhelm any retail stock softwares database and even if you can get past that our tools/heads don't have the data mining algorithms needed to analyize the data. In some sense though this is exactly why I'm interested in this stuff..I read a great quote by D.E Shaw that the best place to find an edge is in places its hard for other players to go.

Well, I mean i guess you could just trade the one with the better sharpe ratio. To me the biggest issue though is only testing on trades in may and june. I think you really need to question if that is a robust enough test as those two months had pretty similar volatility, trend/rangebound characteristics to them. Right now its pretty impossible to say if your system is exploiting a characteristic of the market or simply gets lucky under the specific trade characteristics from may/june.

If you read the "System Developement with Acrary" thread on elitetrader and Evidence Based Technical Analysis, it seems the best way to judge a system with our current tools is to run monte carlo simulations of the same hold time as your systems to see that your system does much better than random. If you were to find that both systems have an edge better than random then you would probly want to check for the correlation between the systems themselves. If they are strongly correlated then you would want to go with the system with the better sharpe I would think. If the systems are weakly correlated togather then you might want to run both at the same time if they have an edge better than random.

To make things clearer what I'm talking about macro/micro wise I would like to return to the box plot, as I really think it could have alot of value for trading. Say you have 5 minute candle data...now if you try to program a strategy what price do you use for each bar in the strategy? If you just use the close your not even using candles anymore but something like a line on close chart...if you average things like what I've seen in some strategies (O+H+L+C / 4)then it would seem to make more sense to use a boxblot. Then you not only have the average price for the bar but also how the prices were distributed around the average. It would also overcome some of the problems with time frames as the time frame is just the sample rate at which your displaying your observations and not weighting the open/close at all. You could still factor in each trade basically without throwing out anything like with candles.

I think this is literally the million dollar question for me. If it takes me 10 years and a phd in bayesian inference just to find out I'm wrong then that is what it takes. My suspicion is that because of the nature of double auction markets that how the bid/ask is being transacted is probly the most fundamental information we have. I would suspect that the big guys who move the markets can't hide their footprints there no matter how much fragmention they use. In some sense I'm starting to view the market price as an illusion since everyone has to transact at the bid or offer. To me something to think about with this plot is if it makes more sense to seperate the macrostructure from the microstructure. I think if you try to include too much microstructure it will obfuscate the macrostructure patterns. This somewhat raises the question of what are the macrostructure patterns that matter? To me the ideal macrostructure plot would be both visually easy to understand while also being easy as far defining patterns algorithmically. Like how a triangle is easy to see visually but a nightmere to define algorithmically using candles or OHLC bars.

I really like the ideas you have here. One nice thing about that is that its different enough from market delta compared with my ideas that you could post it on here when done. Market delta seems pretty up on defending their intellectual property. As far as time goes, one thing to keep in mind is that if you remove time from the axis on the chart its still there but its being represented by the speed at which the chart plots. One thing thats also nice about that then is that a counter/timer in a strategy becomes alot more usefull. If x did/didn't happen during y amount of time is alot easier to program than 10:00 to 10:15. The distribution of the bar to me is the hardest thing to respresent in a way that has meaning. One problem I can see is that if you only have one intrabar POC, then if you have a wide range with 1001 at the top and 1000 at the bottom...it would look exactly the same as if you have 1001 at the top and 1 at the bottom even though that is very different information. If you really wanted to get crazy, the optimal way to represent things might be something like that mock up posted with the vertical 1 pixel line for the bar range and 2 heat maps on both sides for the bid/ask. I think someone asked about heat maps the other day on the ninja site but I can't remember what the response was. One thing I would like to stay away from is value areas. While I love the core market profile concepts to me the value area is worse than wrong. We know that the distribution has nothing to do with being normally distributed so to me thats creating something that is intentionally deceptive and simply has nothing to do with reality. Also, if you want to do this by plotting using the bid/ask it wont be possible to do until next year and the new ninja version comes out. I'll have to figure out how to get at my old drive. I had stripped down some of the plot override in the multiple timeframe ninja indicator. I'll post the code when i get at it. From what I recall once you figure out the cordinates on the chart its not too bad to override. I never really did figure out though how to store things in an array for each bar like what would be needed though.

The math knowledge/programming needed to benefit is pretty extreme.I think the issue you can run into pretty quickly with this stuff is a little knowledge is more dangerous than no knowledge at all. Anyone can buy a commercial neural net retail trading program and spit out data that may or may not be meaningless. Without the theoretical background though I think its pretty much impossible to know if your analysis makes sense. The only reason to use Matlab/R/whatever is that stuff is already there for you to use. You could probly write a monte carlo engine in most trading software that has a robust programming language but its just a waste of man hours at some level. The 2 books that got me going down this path are Evidence Based Technical Analysis and Pairs Trading. I think I did a review of Pairs Trading in the book review section here.

Ok, I've had alot of coffee and now I'm just rambling but... Does anyone know if something like this is possible in any software available? Using discretionary trading, I like how things are setting up using discretionary trading technique X. So I want to get in the market, I hit a button and then my blox box fill algorithm triggers and it scraps for ticks, checks that I'm not stepping into a stupid situation on the order book, a stupid situation as far as the cash vs futures, ect ,ect... I'm not sure what can do this but that would be a very powerfull way to trade and combine algo and discretionary trading.

I really think the key here is to get rid of time as a variable in what triggers the plots. To me an interesting question to start from is... in a double auction market, can price go from 100 to 120 intraday with most trades hitting the bid? I'm not sure that even makes sense since for the market price to climb 20 points someone has to be lifting the offer and getting in at "bad" prices. Thats where I think plotting at X amount transacted at one side of the market makes the most sense because that is fundamentally what is happening in the double auction. Of course no one is going to make a trade decission because its exactly 10:46am as opposed to 10:06 and 21 seconds. The biggest problem though is if you go down this path your pretty much on your own as software/data feeds are not setup for this type of analysis in general. I'm not even sure you can buy tick data that has how the tick was transacted as far as the bid/ask, even though in some sense thats probly the most important information to know.

Haha, thats great. Can you describe this a little more? I think you mean the game "Breakout"? How ironic the game is called "breakout". Also, Intrabar POC/PVP is really nice to have. Market delta already does this by putting a line at what price had the most activity at in each bar. Its really nice to see in a trend, especially for the market profile concept of higher/lower prices atracting activity.

That would be cool, I think I've put things on hold though until the next version of Ninja when they said it will have support for historic fills for delta/bid/ask stuff. Have you done much with overriding plot in ninja? I was starting to make some progress, then my computer crashed and I haven't gotten at my old drive yet. What I really want to do is have something like Market Delta's delta bars but seperate the bid/ask into 2 columns. You can set market delta to only plot at X ammount of transactions at one side of the bid/ask. Its especially nice for trading breakouts because if the market is stuck, bars don't get plotted until the breakout occurs. Here is a mock up I did a few months ago, and another one with a 3d order book thing but I don't know if that would really be usefull.

I don't know, Fooled By Randomness is one of my all time fav books. I've tried to read The Black Swan a few times at Barnes and Noble but just can't get into it. Another book that is possibly better is Plight of the Fortune Tellers by Riccardo Rebonato but I haven't read it yet. If you look on the quant boards they all think Taleb is a tool but respect for Rebonato.

I've started working on writing up a boxplot in Ninja, i'm actually kind of suprised we don't have boxplots in software already. To me its not an issue of finding new data, of course your not going to find something that is not there. To me the issue is what is the best respresentation of the data that is not either ignoring some of the data or weighting some of the data more than others giving a distortion to the data. If you take traditional candles for instance, there is somewhat of an arbitrary overweighting to the candle body depending on what time frame you decide on. To me the order book is the big thing in the equation that we are missing, looking at the raw order book and just calling it nosie to me is a lack of imagination as far as a descriptive summary goes...although I can't really think of anything usefull in this area either.

Data is probly the biggest pain at our level for this stuff. You could probly buy what you want from http://www.tickdata.com. I think at some point though your going to have to store your own data and most the software available to us isn't really built around storing massive amounts of tick data. The only real off the shelf solution I can see is open quant or neoticker, then export your data as a CSV file/files for analysis. Even with those 2 i'm not sure at what point you would overwhelm its DB capabilities though. At some point I think you need to bring in your own time series database. This is where matlab would probly fit into things best as there is a toolbox for connecting datafeeds to it then most time series database would easily hook into matlab. The big problem there though is the datafeeds supported jump right from interactive brokers to bloomberg/reuters without esignal/dtn in the middle.

From what I've found I think its best to go with R starting out. The biggest advantage is there are tons of text books on how to do specific techniques with R so you can both learn R and the technique/theory at the same time. Matlab is much prettier but you would more or less have to learn from purely theoretical textbooks(or textbooks with code in R) and then translate it over to matlab. For me at least the hands on experience with R is much better without the structure of a class room as I go. R can also be called from matlab or excel too so even if you needed better GUIs someday your code/knowledge isn't going to go to waste. Programming straight out in VBA in Excel is surely an option too but I would think you would run into bottlenecks sooner than later. The program you pick probly doesn't matter much though, the real issue is how feasible and driven you have to be to learn graduate level machine learning and stats on your own. Also MATLAB with the toolboxes you would want will run several thousand dollars, R is totally free.

Good stuff, I'm undergoing the same process of converting over to a "baby quant". Modern data mining ideas have virtually not sifted down to the retail crowd at all. I don't really see this changing either. I mean its easy to sell the retail crowd neural nets because everyone has a brain and then assumes they have the background to understand a neural net conceptually. Try to sell the retail crowd a support vector machine product and its obvious there is alot of background information needed just from the name of the technique so no one would buy it. To me as far as banks go the question is what techniques can we take from them that will be good at our level as opposed to how you can trade like a bank. Data mining and time series analysis stick out like a sore thumb IMO. I also somewhat believe that there is a missing concept as far as risk goes. To view the risk of a 100% drawdown on a 5k account as the same as a 100% drawdown on a 10 Billion dollar hedge fund seems pretty absurd. Some place in there there is risk of not putting on enough risk in the 5k account. To view things the same as the way people handle the risk of having to raise a billion dollars doesn't strike me as optimal.

In theory of course they have enough capital to blast the futures but in practice they wouldn't. All they would be doing is giving their money away to their competition. If Goldman started blasting YM all the other big guys arbitrage programs would start blasting them in the other direction and Goldman would just be giving risk free money to the competition. Personally, I think "manipulation" is the most overused word in retail trading. Trade volume doesn't break down to Us vs Them...it breaks down to Them vs Them...retail guys are less than 5% of the volume and mostly just in the way of 2 huge sharks trying to eat eachother.

I'm not exactly sure what you expected people to post here to the original question. People have told you exactly what is going to happen. You go prop and your strategy will no longer be secret. I don't care what legal documents you sign. If you notice you don't read about hedge funds tied up in court all the time because hedge fund A hired away hedge fund B's hotshot and traded their ideas. Hell, I'm sure the main reason managed funds have thrived to such a degree is that things aren't bogged down by all the insane intellectual property laws. If your strategy is X, once its no longer secret I'll go X - a tick..there now its my strategy and the legal document you signed isn't worth the paper its wrote on. A system that produces numbers like that crushes every CTA in existence. Prop will get you up and running and some money in your pocket at the expensive of giving away 10s of millions of dollars down the line with your own CTA. Thats why people are saying not to go prop.

I think this is pretty much impossible to answer. I mean at the fundamental level the indexs are correlated because its a snapshot of the markets pricing economic activity forward. Then above that you have alot of shared components single stock wise between the indexs. Beyond that is when things get murky IMO. If Goldman is going to build a position in IBM maybe they sell big S&Ps to hedge the position, that then causes someone to arbitrage the big S&P vs the ES, then that causes someone else to arbitrage SPY vs the ES, causing someone else to spread the ES vs a basket of DOW cash, ect, ect...All which is happening in a non linear fashion, probly hundreds of times per TICK. I don't see how we can really gain any edge at the retail leve from this information. To me the one possible edge that might be there is to calculate a tick precise PREM client side, server side being sent through your data stream seems to slow. Then figure out on your time frame what leads what statistically, cash or futures and then only enter trades when the higher probability situation is giving you a few ticks as opposed to eating them.

I kind of have a speculation that if you look at the shape of the intraday volatility surface of SPX options and how its changing that it will give away exactly what kind of day to expect on the futures because the guys that move the markets are too big and the options markets too illiquid for them to not show that card. How to actually view that card, I haven't a clue yet though.

To me you have to bring in how these instruments that we are trading are used by the big guys that move the market. I mean how much of the daily volume on index futures are the big guys hedging? Its fully concievable that alot of times the futures are being sold while the participant selling the futures are expecting the index to go up because they are buying a basket of the underly. To me its a very important distinction to think about. That at the retail level we tend to use derivatives for leverage to add to risk, while most the volume on derivatives are actually to reduce risk by the big participants. In some sense though it still may be a non issue as far as volume/auction analysis on the futures. While at the micro structure level things are way off but there is enough activity that at the macro level the futures are a perfect replication of the overal market auction process for the day/week/whatever.

Your spot on there. I've entered a few competitions like this and the difference between first place and last place is always that first place has one extremely positive outlier while last place has an extremely negative outlier. If your trying to win and there are 20+ people in the class your best strategy is to put on 10 of the highest risk, correlated bets you can find and pray. Obviously, if your being graded on building a real world/real money portfolio then this doesn't make sense. If your just trying to win though diversification is the enemy.

The first sentence on wiki as far as randomness to me sums things up perfectly. http://en.wikipedia.org/wiki/Randomness "Randomness is a lack of order, purpose, cause, or predictability in non-scientific parlance. A random process is a repeating process whose outcomes follow no describable deterministic pattern, but follow a probability distribution." I think the difference between randomness in the general way the term is used vs a randomn process causes alot of confusion when ideas drip down to the retail level from the academic world. To me its hard to come up with a better definition of the markets than "The markets are a repeating process whose outcomes follow no describable deterministic pattern, but follow a probability distribution."