darthtrader2.0

While I will probly forever respect this thread on here for motivating me to be non igorant of statistics and probability, from what I've learned in the past year most these intial posts are pure frequentists nonsense with holes you can drive a truck through. Essentially, this entire series puts forth a very scientism view that between the high and the low of any given session, one can use a probability density function based off how volume is distributed dependant on price,an idea that already assumes a normal distribution to the way that fundamentally, we already know that each trade is a stochastic process, and the sum of that stochastic process has nothing to do with a normal distribution as far as price or number of trades at a certain price goes. Let alone as far as the stochastic process that each of the participants information that they base their decisions on as far as a single unit of trade, just more NULL here. What I would suspect is that this system works for you absolutely because of the unknown variable of your understanding of price action and that all this nonsense, IS your subjective bayesian filter for your OWN understanding of "price action", it certainly has nothing to do with the bunk science of frequentist statistics who can not predict shit. Touché

No offense but this is all nonsense IMO. In a low volatility range bound market fading the programs near the top/bottom of the range makes sense. This is the regime on which most stuff that has been written about TICK is based but that situation is long in the tooth. You have to account though at what interval your datafeed is giving you the TICK or your not really seeing what the programs are doing. Sadly, it only seems that tradestation is reporting TICK at 1 second intervals? I know DTN certainly is not, anyone know if etrade is at 1 second intervals for internals? From being obsessed with the TICK and how the program trades are sweeping the markets, it strikes me that we have terrible representations of the underlieing data. Its obviously a very noisey data series and I dont see how candle patterns have any meaning in this context. I would love to have a conversation on how to best represent TICK on a second by second data feed. The best I've come up with(keeping in mind that DTN reports tick every 5 or 10 seconds, i forget what) that is perfect in hindsite but useless in real time is to use ninja with DTN to take every TICK reading, sum it, then divide by the number of TICK readings and plot the result giving you a very granular cumulative TICK(on DTN it gives you essentially an 8 day tick by tick moving window of the cumulative TICK). Randomly some days it is meaningless(or in other words the programs aren't doing much that day), somedays its the most deadly thing I've ever seen plotted, but only in hindsite(which I currently atribute to the interval of DTN reporting but would love to be proved wrong there).

No offense AgeKay, but you are totally biased since your a real programmer and have your own self interest in this debate.(self interest is not a bad thing to me so please don't take it that way) To me its like if you hired a team of programmers to make you the most baddass software on an apple 2e in 1984..no matter who you hired that code was virtually worthless not enough years later to justify the expense. Probly a better example that if you had hired the guys that brought neoticker to market for your own custom software, at this point its such a mess it may as well be scrapped, save the concepts/IP and pay someone on the cheap to re code those ideas in C#. Basically, this business moves so fast robust software simply does not make economic sense vs a hack. You have almost virtual certainty that your robust software will be clunky 5 years from now. The hard part of any automated system at this level is the logic of the system itself, the software engineering aspect is somewhat nill. Maybe I'm biased because I would be interested to see how a computer science student on the cheap implimented my logic from my lack of experience in software engineering but even if they went out of the way to make the code unreadible, its still my logic that they are expressing. I forget at what point I heard about the concept of "fast, cheap and dirty" as far as software engineering goes(or something like that). I think it was some AI nonsense during the dotcom bubble...That concept certainly doesn't apply to AI but to engineering trading software ideas, that seem to me to be right on the money.

Well it sounds like you are starting out on the right track. I totally agree with finding the best representation of what is happening. I'm also using DTN/Ninja, hard to beat that price wise and Ninja's tape is the best hands down. My current setup is 200v candle chart with the wicks removed so its just a "body" between the high and low of each 200 transactions and the tape on YM. My setup is to trade triangles on that chart and then entry/exits are managed on the tape once the triangle is broke. After 4 years at this, that is the only setup that passes my soft subjective confidence interval without backtesting but since those don't happen very often most the time i'm just sitting and waiting looking for other setups that look nice to trade someday. A good excerise I'm doing while waiting right now is to draw extended horizontal lines in Ninja at where you believe the market will respect a price and then see what happens on the tape when price dances with that level. The funny thing I've found is no matter what I draw, the only thing that seems to have better than random results is the overnight high/low and the session high/low. The bottom/top of an intraday range will be respected on the tape once then mean nothing the second time(all that information has been discounted), or be respected 3 times then mean nothing the 4th time with zero predictability or soft expectancy. Currently, the only little pattern I'm considering trading beyond my triangle position trading is a scalp off the tape. That if you look on a market delta chart, the top/bottom and cascade of a move is always marked by extremely "aggressive" traders that are totally in the wrong direction. On the tape this looks like, if you move your perception up/down a bit from the current action to where you don't believe any orders should transact, you will see a few prints at those levels and then price reverses. The problem with this trade for me currently is that I've also seen thats when the programs come in and sweep the market sometimes and my trading plan #1 rule is to never put myself in a situation when I know I could be on the wrong side of the programs. The problem here is that DTN blows for internals and I might park into a tradestation account just to get their nice internals.

Good stuff, I'm not really sure it matters to think much about what the huge banks are doing as far as stuff like this goes. If you know about it, they are probly doing it and then some. I'm not at the level of capital yet or skill with my single strategy that I have to bother thinking much along these lines but I do think you can gain some utility from thinking about this stuff from an autotrading standpoint. The value ultimately would come from a probabilistic sense of that if you have 2 different strategies with positive expectancy, if they are both trading against eachother you will get a hedging effect as far as risk goes, then when they are both on the wrong side that situation will be less costly than the magnifying effect the 2 strategies will have when both on the winning side. While I hate elitetrader, if you read all of Acrary's posts there is a goldmine of information on this kind of thing. I'm not sure it makes alot of sense though to be putting on 2 strategies until you have mastered one. I mean if you have any advantage starting out its ultra pure liquidity. You would probly be better off using various entry/exit methods on a single strategy than viewing one strategy as a core and then another strategy against it.

If you use filters your going to be cutting off alot of orders that are big money that just has chopped up their orders. I agree with throwing the books in the garbage. Probly the best way to start out is just watch the tape when price nears the session high/low on a thinner contract than ES. Also its probly a good idea to have TICK open so you can see what happens on the tape when the programs sweep the contracts and that the size is usually in the speed of small orders as opposed to very large prints. If you just start with the high/low though you can get a feel how the order flow moves if the high low is going to act as support/resistance or a breakout from the high/low. Also, just take your time and don't get frustrated.

Yea I dont know, I got it this week but have barely opened it. Ive really broke through some barriers knowledge wise in the past few weeks and honestly, I think once you get past the pure microstructure trade mechanics most of this stuff is a bunch of nonsense. I would love to compare notes, here are my mental notes as far as high frequency trading and microstructure: First, I think this whole business suffers from this concept of "phd-ism" that I have. Somewhat of a game theory concept where it matters far less that what you know has utility than if you can make the opponents in the game believe what you know has utility. Objectively, you are far better off in the game of life if you can make everyone believe you know something that is bogus than if you actually knew something that is true but no one believed you, phd-ism. Thats somewhat how I view all these microstructure ideas when you really cut to the bone.We are talking about a game theory model with X participants, x == the number of individual traders making trades over Y time...the variables at the high freq level are not just unknown, they are "unknowable" for the sum of x so the market can not be truely "gamed". My interest with this stuff though is that given enough data, time and knowledge, can you "game" the market better than 50% of the time. To me that is an obvious yes, or no one would be able to make a living as a trader.

ehh, damn that was fast.. I was going to try to build an arguement towards constant volume charts but I guess its always good to cut to the chase. To me the problem is no different on volume charts than with the previous time based example as far as respesenting each tick. The problem for me is that the body of a candle is formed by whatever time frame or volume level you decide to chart it as. Just because software developers decide to make the defaults 1,3,5 minutes or 100, 200, 1000 volume wise does not mean that is the optimal way to summerize the underlying tick data. The biggest variable in what patterns are displayed in the summary of the tick data depends on what time or volume variable is chosen...if it makes no sense to display a 4 minute and 39 second chart or 527 volume chart, then to me you need to prove that a 5 minute or 1000 volume chart has utility over the former or your simply accepting randomness in your analysis that could be cut out. How to cut out that randomness I do not know at this point, but to me that is a very interesting discussion.

See this is actually why my brain refuses to get into candle patterns. If one unit of representing data does not have meaning but 3 do...then there should be a better way of representing data by combining what gives those 3 units meaning. To me a good example is the good ol doji on a 5 minute chart. Instrument X opens at 100, volatility is high for the day so a massive spike on the NYSE TICK sweeps instrument X to 105, 30 seconds later price falls back to 100 and drifts down over the next 2 minutes to 95, then slowly drifts back up over the next few minutes to close near 100. Another day Instrument X opens at 100, volatility is low for the day. Instrument X opens time wise 5 ticks below the top of a range, price sits between 99 and 101 for 4 minutes, the test of the top of the range comes quickly after 4 minutes, price is rejected down to 95 quickly...buyers step and defend the top of the range and move price back to 100 over the next minute..bar closes, 20 seconds into the next bar the range breakouts and price blasts to 110. Both are dojis, both are totally different information wise. Obviously a skilled candle trader would pick up on the totally different context of the doji in those two situations, the newb would not. What I question though is that there is not a better way to summarize the movement of the underlieing ticks that make up both dojis, considering that if you changed your chart to a 1 minute candle chart, you would see a candle pattern and not a doji, and would see a different candle pattern in each instance.

Ive honestly never found any utility in the delta volume indicators. The only thing that seemed to be pretty usefull IMO is Market Delta with the plot set to delta bars. They are basically like constant volume bars but a new bar is only plotted when the bid/ask has transacted to one side by a number you set. One good way to use that is if your in a winning trade just hold until the move has reached its top but you have the opposite as far as delta goes. Its a good bet that those who were lifting or sinking the bid or offer of the move have played out their hand and some profit taking has come in. Its too inexact IMO to use for countertrend reversal trades though.

Well I would say you should really decide now if you want to trade your own capital or be like one of these guys. If you want to trade your own capital then a CS degree would be ideal, I mean at somepoint your going to want to do some programming so you may as well learn the nuts n bolts while getting a degree. If you want to be like these fund managers then CS would be a bad idea since it will probly push you towards being a software developer for these guys as opposed to one of these guys. Also you should be aware that economics sounds far more interesting before your in the class then when your taking the class. I would imagine if you sat down and digested Keynes and Hayek's two master works you would probly understand economics far deeper than someone with a BS in the subject. Kind of like with pyschology. I would imagine if you had an in depth conversation with 500 different people you would understand pyschology better than someone with an undergrad degree in the subject. It really depends what you want to do though as you can't put # of conversations or memorizing The Road to Surfdom on your resume.

Well I've finally crossed a milestone in the past 2 weeks in that I can actually tape read. I'm not sure though that all that "stuff" didn't advance you towards your goals. I think the biggest problem with trading is you don't know what you need to know until you know it. 6 months ago I would have laughed at myself if I thought what I've been looking for all this time is the tape with a simple constant volume range bar chart. There have been points where I started wondering if people were actually lieing or deluded when it came to the tape although I do remember someone mentioning that Market Delta is useless if you can tape read which I can totally understand now. While all the books and information, thinking about getting matlab to datamine ticks(basically tape read) seem like a bit of a waste of time now, I don't think I could have ever arrived at this point without learning what I dont need to know first. While I know brownsfan rips it up with candle patterns, I think If I had started with candle patterns and never bothered to explore everything basically, I would probly have quit by this point no matter how much screen time I put in. Candle patterns just do not mesh with the way my brain operates on any level. I heard a great quote on an economics podcast that seems to fit along these lines..basically from probability theory, "to increase your chances of success, double your failure rate". Basically try everything and fail at it and eventually you will find what works for you.

Well I'm sure you can use the tape to get better entry/exit price on your swing trades. That might be trying getting too microscopic though since if your scrapping for ticks on a swing trade its not going to matter much if your totally on the wrong side of the move. Something I've found with the tape is that I'm not sure if alot of the older literature on the subject has been degraded value wise by fragmentation algorithms. If someone executes a 100 lot YM order I think you have to question who that trader is...in the past that probly would have been the biggest guy "in the room" but at this point the biggest guys in the room are certainly going to be chopping orders up and not showing their hand to that degree.

Thats what I mean though as far as I think it depends on the robustness of the software that you need. If you want someone to write a completely charting/order package then I can totally see paying a pro to do it. If your really just wanting a data I/O hack then I would think you could just get this cheaper without paying the experience premium. As far as computer science students, while sure most will be in the program just looking to get a good career, I'm sure every crop has a few real geeks who are there because thats what they love to do. To me the real advantage of this is for every hour you pay that type, you would probly get 2 extra man hours for free of work. When I'm at brownsfans level what I will be looking for is that guy when they are a jr who also plans to get a masters/phd in data mining/AI. If things worked out you might have an employee on the cheap for a few years. The cool thing with that to me is the projects you would be offering would be far more interesting than the stuff in school so you could probly get free man hours and especially get man hours that would not be economic to pay a professional data miner.

For me I've just found that trying to find balance is a waste of time right now. No time for a girlfriend let alone a wife/kids, work a lesser job in the evenings than what I could get when the market is open, a piece of junk car, ugly old house with cheap property taxes...I just view it all as cutting expenses to improve the balance sheet while not limiting the opportunities for massive growth. I'll find balance when I get to brownsfans position...even if it takes 20 years to get there, for me this is much more interesting and fulfilling than 10 hour days doing some mind numbing financial services job.

just to throw my possible worthless 0 cents in here... I just have to question if something like this is worth paying a real software engineer to create robust software instead of paying some computer science student an 1/8th of the project budget and just have them hack something out. I'm probly biased because I live within walking distance of the computer science lab at the university here but someday I totally plan on posting a job add. Then interview to find the most hardcore geek who will put in way more man hours into the project just because they enjoy doing it and whatever I pay them will be way more than working in the library. Not to mention they would also be paying themselves by have something to put on their resume instead of work on a free open source project. Right now especially, I would think guys would be frothing at the mouth to get C# experience. While of course you would lose out on design cycle experience, I just have to question at what point that extra cost makes sense if your just trying to I/O data and do some pattern matching. Not to mention, the softwares life cycle may be far too short to bother with robustness...maybe 4 years from now you find something better than OEC and your back to square one with the API issues. Also on a tanget..I'm really surprised no one has stepped up to the plate and created some kind of universal abstraction layer software that would act as a bridge between any data vendor/broker and any software behind the bridge. While I'm sure this would have pretty big startup costs, at some point I would think it would gain economy of scale that vendor/brokers/software companies hand would be forced to have to support it and at some point pay a licenseing fee. The API issues in this area are just absurd in its current state. Everyone is doing something slightly different and 3rd party software guys dance through hoops while everyone is trying to do literally the exact same thing.

Well he is not a hedge fund manager but he has posted in his blog that he only trades the morning session mostly because it frees up the remainder of his day for his other pursuits. In a business loaded with such charlatans its hard to knock a guy who gives away so much valueable information for free. To me an edge in trading is the sum total of what you do that beats the monkey throwing darts at a dart board. Of course this doesn't need to be backtested, you can forward test what your doing purely discretionary but you may not realize you have no edge until you blow up. I soppose the same thing goes for backtesting too in that you may not find out that your "edge" was just data snooping bias and that you didn't account for enough variables until you blow up.

econometrics & statistics would actually be your major? That is pretty cool, you wouldn't really get into that level of specialization here until grad school. Keep in mind though that econometrics may be the most boring subject man has yet to create..it should certainly give you a good toolbox to work with after school though. I would be pretty surprised if they don't have a few programming courses thrown in with that program.

ack, you clearly understand something in Harris that I do not. You really should make a microstructure thread and spill the beans...I want to be fascinated by this subect and then maybe I can possibly pay you back for such a thread with my own "take" but I'm totally brought to a stand still by the language. The idea of a utilitarian trader simply does not compute on any level to me. How about informed trader == front runner? How off is that? I know I have a weakness with digesting theory if I can't see how it fits into a pragmatic strategy, even if the theory makes total sense.

Hahaha, gone to town with your Yen/S&P correlation ideas, ehh canadian? It actually is rather new, I haven't jumped into Gatheral's The Volatility Surface because I'm still gaining the mathematics to understand it. However, I do believe you are correct, there is this 3rd dimsenion of volatility we don't talk about that is "gamma-ish", but whatever you want to call it its more 3d and overall across strike prices than the gamma of a single strike.

Well first, very impressive thinking on your part overall in this post. To the above, its the effect of networking plain and simple. The quality of university is extremely overrated as far as actual knowledge gained. Most of the benefit is purely political, if someone takes calculus at MIT or community college obviously you view the person who took it at MIT better...however calculus is calculus. Its not as if there is hidden basic calculus knowledge at MIT. As far as age, comeon...you are way better off judging from this post than some 18yo rich kid moron who goes to Harvard because he has to keep up with what mommy and daddy did money wise. If I was 20 and from the land of oz I would get a degree in a math, statistics or computer science and get really passionate about whatever the biggest, hardcore competative sport is in australia...rugby? I don't know...Whatever that sport is it will give you the political edge on the opponent for a job, even if they went to a better school. For your own real knowledge take some game theory, programming, and bayesian probability classes if offered...certainly don't waste your time with an economics degree.

Yea I don't know, like I said, I have a real philosophical problem with the informed vs uninformed trader deal... I have to deal with these idiots everyday at my "day job"...I can assure you, no one cares the least about this stuff....its just a "job" with the possibility of making "alot of money". I've never met an institutional trader who was not some sports nut who wants to talk football instead of markets.There is no asymmetric information on an index that is not priced in IMO...

Well first off, are you Dee from Stocktalk? To answer your question the "programs" are evaluating NQ at the millisecond level...not minute to minute... I do agree with you overall though...basically at the retail level we negate the 3rd dimension of "volatility". Your simply giving that dimension a name. This idea though is something that hardly has not been studied. Just google "volatility surface" instead of reading about useless 2D "volatility".

Well, I've tried the past 2 nights to read some more of Harris but I give up for now. While its loaded with information I have to say I hate the writing style. The whole thing seems kind of scatterbrained and the sidebars kind of drive me nuts.Like the volatility chapter...just when it starts getting good the chapter ends and moves to another concept. I think another problem I have with it is I have a real problem with the whole concept of informed and uninformed traders. I found some sample pages on google books of Empirical Market Microstructure and it refernces Harris as a broad overview but references Market Microstructure Theory by O'Hara as the standard reference. I'm going to move on to that and pick up Harris later. This microstructure book sounds baddass too even though it doesn't reference microstructure in the title: Asset Pricing under Asymmetric Information: Bubbles, Crashes, Technical Analysis, and Herding by Markus K. Brunnermeier. Here is also something from Empirical Market Microstructure that to me makes alot more sense than the concept of informed/uninformed traders: "Liquidity is created through a give and take process in which multiple counterparties selectively reveal information in exchange for information ultimately leading to a trade."

Wow that is a kick ass site. It might be worth considering paying the 100 bucks for the extra web content they seem to be offering. Thats a pretty interesting idea as far as looking at time stamps all at the same bid/ask. Considering the fragmentation algorithms are all executing in the milisecond range. Very impressive with the amount of homework done on that site plus the fact that the software offered is not priced "to move".