BlueHorseshoe

Hi Predictor, Thanks - some really useful ideas there. I think I would agree that it's possible for very well capitalised participants to push the market around by a few ticks or so, although probably not usually be more significant amounts. There are no designated MMs in the ES, so it's always hard to guess whether anyone unofficially assumes this role. It's always easy to say that there is no need for MMs because the market is always so liquid, but is it only so liquid because of those who discreetly act like MMs? Personally, I would guess that there probably are HFT firms trying to occupy this space, although how successfully it's impossible to know. Your point that if a firm holds most of the book they can more accurately estimate position is a fascinating one. It means that if such a participant is already queued at the time one wishes to join the queue, then one may be at a disadvantage in terms of estimating one's own PIQ thereafter when compared to them. It would also provide an additional motive for things like trying to re-construct icebergs. Thanks. BlueHorseshoe

One thing to remember is that we’re only really interested in reductions in the orders ahead of us in the queue. This means that we could potentially treat all other changes as ‘noise’, and then try and filter the noise out to reveal an estimate of the underlying time series. Something like a Kalman Filter could be used, or some sort of Neural Network. The problem with these methods is that they both require the back-propagation of error. This essentially means that a ‘prediction’ is made about the subsequent state of the order queue ahead of us, discounting for an historically optimal degree of noise, and then the prediction is compared against the actual output at the future time, and is used to correct the weighting variables within the algorithm. Here we face exactly the same issue as was described in my previous post, namely that there is no reliable description of the underlying state of the order queue. Not only can we not derive it ‘live’, we can’t even get at it with the benefit of hindsight! This is very frustrating! Imagine you were allowed to send orders back in time to trade last week’s markets, but you don’t have a chart of last week’s markets, or any information about last week’s price movements. What do you do? The only other time that we can be sure of our error is at the time an order is executed. Theoretically, data could be collected in this way and then used to train the algorithm. One problem with this is that it would still only allow us to map the state at two points - entry and exit. Another is that it would be expensive. A round trip in the ES would be required just to provide two data points. Not much use . . . BlueHorseshoe

My apologies, MightyMouse - you caught me in a bad mood there! I am currently working through a series of questions relating to liquidity, orderflow, position in queue, and efficient execution. I haven't made use of any of this in my trading before as I traded from daily charts and found that execution had a fairly negligble impact. I'm now considering day-trading strategies, where I have become convinced that a thorough understanding of all the above would be very helpful indeed. If you want a specific example of how then I'll provide one. Hope that answers your question. BlueHorseshoe

Understanding liquidity around a price means that you can make better judgements about how to execute your own orders and, depending on your goals, whether to execute orders. Coming from you, I'm assuming that you're asking this question for the benefit of others? BlueHorseshoe

The value of this information is not in predicting price movement, it's in knowing relative liquidity around the prices you trade. BlueHorseshoe

I'm not interested in "common sense", mate, I'm interested in what is actually the case. It's actually the case that in Post #19 you state: . . . when in Post #3 I had already stated: Within my platform order book data cannot be collected historically; it is only available to be streamed and then stored at the time it is created. Therefore the data that can be used to backtest is the same data that would have been available at the time of trading. Can you see how you're continuing to talk drivel here, "Colonel"? Don't trade in sim; don't trade intraday fella - now how do you like them apples? Yes, thanks for the basic definition of volume. If you read post #3 you'll see that it's perfectly clear that I know what volume is. Not sure why this should even be a point of doubt. Call it what you like (I never look at it, so I don't care). It doesn't change the fact that the statement above is completely incorrect - the orders on the ladder/DOM are the queue/depth of book. No "Colonel", I'm angry because you explicitly claimed that nobody had stated something (suggesting that everyone else in the thread was perhaps a little less enlightened than you), when in fact I had clearly stated it in the third post. No need to get up early, I live in the UK. Nor do I understand references to "the Bush league team" My point still stands, "Colonel" - you made a claim that was incorrect regarding the contents of previous posts in the thread, and you claimed that certain data was not available for backtesting when in fact it is. Now when are you going to admit that you made two incorrect statements in your last post? BlueHorseshoe

Yesterday I posted two games. In the first of these it is easy to see that each of the magenta symbols corresponds with a particular cyan symbol, so that the final four missing symbols in the cyan sequence should be: cannonball, skull, hourglass, cannonball. If I could identify a static and certain relationship between what I am trying to model (PIQ), and another discrete time series (eg Price), then it would be very easy to use the one to 'fill in the gaps' in the other. When I became uncertain of the PIQ value, I could simply look at price and this would tell me. Unfortunately such static relationships in market data are incredibly rare. A little more common, however, are relationships that hold for most of the time, where two or more time series are correlated to some degree. In the second game we can guess that this is the sort of relationship that exists between the magenta and the yellow sequences. We can't know for sure what the final four symbols in the yellow sequence will be, but we can make a pretty good guess. Two out of three times, omega corresponds to square; three out of five times, paper corresponds to circle; three out of four times yin-yang corresponds to diamond. P(if omega then square)=0.66 P(if paper then circle)=0.60 P(if yin-yang then diamond)=0.75 The final four symbols in the sequence will most probably be square, circle, diamond, square. How useful is any of this in solving the PIQ problem? I reckon probably not very. . . . If there were sections of the PIQ series that we knew for certain, and we were able to find time series with which they were reasonably correlated, then we might be able to model the missing sections. Unfortunately though, pretty much all sections of the PIQ series are hidden. There are only two times that I can think of when we can be remotely confident about the actual PIQ. The first is when we place the order, and the second is when the market ticks, the depth remains unchanged, and the depth is fairly significant. Suppose that there are 99 bid and we join the queue, making it 100 bid. The market ticks and there are still 100 bid. It is entirely possible that though the depth remains unchanged at 100, 73, say, of the original orders were cancelled and 73 new orders added, so that our PIQ is now 27. Assuming the probability of a group of orders of any particular size being cancelled is identical, and that the maximum number of orders that can be cancelled ahead of us 99, then the probability of N orders being cancelled is N/99. The probability of N orders being added is a bit strange because the only obvious ceiling to the number of members of the set of all N is X, where X is the maximum number of contracts that can be traded. For the ES, X is 2000 (Page 15); for other exchanges the number may theoretically be infinite. So, the probability of any particular order size being cancelled, and an identical order size being added at the same time, in our example, would be (1/99)*(1/2000). That's not even worth calculating - let's just say that we can be pretty much certain that when the depth remains unchanged it is because orders have remained unchanged. Although the depth regularly does remain unchanged, this still doesn't help too much with our problem. If you return to the first game, it's a bit like having all the cyan skulls marked in to correspond with the magenta hands, but none of the other symbols provided. You'd have no idea what a crucifix corresponded with because you'd never even have seen an hourglass. The next thing I will try and think about is situations where relationships exist between events in the time series. In the context of the games this means that a symbol is always dependent upon those that preceded it in the sequence. BlueHorseshoe

An interesting link here, not that they're giving much away, of course: System and Method for Estimating Order Position - Patent application BlueHorseshoe

When you say 'scalp' . . . ? BlueHorseshoe

Really? Just as I thought it was just starting to get interesting! BlueHorseshoe

I like that! BlueHorseshoe

The rant about my father was just a joke, by the way That's right. I guess it's not a million miles from the options ideas that some have suggested, as the estimated MAE might have something in common with the implied volatility of an option. Although I don't know anything about options. There are also the obvious drawbacks that would apply when one uses MAE in any strategy. It's a "worst case" drawdown, and maybe if one knows the future it feels a bit too risk adverse? Maybe if prices never fell more than a few points from entry I would suddenly regret not doing more to capitalise on the opportunity by assuming a little more risk and leveraging my position more? On the other hand, there is always a worse MAE lurking somewhere in the future . . . BlueHorseshoe

Hi John, Thanks for the reply. I don't really have an opinion on the point DB is making with regard to the time limits a trader should give themselves. But I do think that the wealth of indicators, books, courses, instruments to trade etc can be as much of an overwhelming hindrance as it can be a help. A lot of the time sifting through the 'resources' that are available is like searching for a needle in a haystack. Given what I know of DB's approach I would hazard a guess that he might agree with this. If there was just one market to trade futures on (just the ES, lets say), and nothing else, and there was absolutely no possibility of any outside input from 'gurus' etc, then learning to trade might actually be a whole lot easier. Just a thought! BlueHorseshoe

Panic would set in. As the market ran away and left me on the sidelines I would begin to feel the sweat running down the back of my neck. And then I would hear the disappointed voice of my father in my ear, berating me thus: "Even when you have messages from the future you still manage to screw it up! What's the matter with you, boy? Can't trade, even when you know the future - you'll never amount to anything! Now go and fetch my beating stick, you useless waste of space . . ." If I used a market order, why wouldn't I be filled? BlueHorseshoe

A fact that can either help or hinder, surely? BlueHorseshoe

I think my strategy would be to use the very maximum amount of leverage that I thought 'safe'. So, if the newspaper told me that prices would be higher on 20th Sept than they are today, and 20th Sept is (I can't be bothered counting) 20 trading days away, then I might examine all instances of a close that is higher than twenty days prior, calculate the maximum adverse excursion that would have been endured if holding a long position throughout each of these periods, use the largest of these MAEs plus a fraction as an estimate of the worst drawdown that I might have to endure, and then postion size a long entry such that should the greatest historical MAE plus a bit occur, I would have sufficient account margin to maintain the position. In other words, I'd go "all-in" using an approximation of what might happen between the two known certainties ('now' and 'then') to control risk. Is that the right answer? Is there a prize? BlueHorseshoe

Colonel B, If you go and read post #3 you'll see that I clearly talk about referencing the bid/ask volume against the queued orders (I even put it in bold). The notion that you can't backtest this is also absolute rubbish - of course you can backtest it! Your data provider will provide a way to stream the queued orders at each price level. I know you probably think that 'once they're gone from the DOM they've vanished forever', but they haven't. They are there for you to analyse after the event, just like price or volume. If you don't know what you're talking about, and also can't be bothered to read what other people have posted, then you should refrain from making authoratitive sounding statements on threads like this - you'll just cause more harm than good. BlueHorseshoe

Backtesting allows you to test a specific hypothesis on an historical data set that may hopefully be representative of future market behaviour. Before you can arrive at a specific hypothesis you need to derive more general tendencies from the data. "I can profitably fade any price cross of a 4 period SMA back in the direction of the slope of a 200 period EMA when the ADX is below 30" . . . Is an example of a specific hypothesis that one could backtest. "I can profitably fade small countertrend movements back in the direction of a longer term trend when the market is showing a lack of trending capability in the intermediate term" . . . Is an example of a conjugation of more generalised tendencies which, if generally held to be true, might inspire a specific hypothesis such as the one above. So the first thing that I would do is to mine the available data to try and establish some general tendencies that you can build upon. Once you have these you can begin to select more specific tools (such as indicators) to exploit these tendencies within a back-testable strategy. Most back-testing software will allow you to carry out a data-mining styled exercise. Here is an example of the sort of thing you might like to know: Do most short term highs occur on higher or lower volume? The trick here is that you'll need to test across a variety of definitions of short term high. These might range from a three day high to a twenty day high. You might find that 100% of all 17 day highs in your test occur with volume that is higher than the previous period. Unfortunately this isn't much help - what you need to know is whether short term highs in general occur on higher volume than the previous period. For all your tests, then, you might average out the percentage of short term highs that occur on higher volume. If this percentage comes out at, say, 78%, then you might conclude (assuming that your data set is large enough and sufficiently representative) that this general tendency will hold to a reasonable degree regardless of the specific periodicity reference used for the high within the strategy you subsequently devise. Hope that helps - anything I was unclear about then just ask! BlueHorseshoe

In the second game the aim is to complete the last four steps of the yellow sequence using the magenta sequence above. BlueHorseshoe

Last night, to help me think about this more clearly, I created some games. I'll post them on here, and if you have five minutes to kill as you wait for something to happen on a chart, then you're very welcome to play them and post your answers. They're all very easy. In the game below you just have to use the magenta sequence to complete the last four steps of the cyan sequence below it. BlueHorseshoe

The word is generally used in two different ways. Is there not something in the dictionary section of TL on scalping? BlueHorseshoe

I couldn't find them listed in the globex manual - can you give me a pointer to more information on this? Cheers, BlueHorseshoe

I've only given any consideration to markets that maintain a single tick spread, so what you describe is a useful thing to think about. BlueHorseshoe

There are plenty of traders making plenty of money who never look at any of this stuff. If the average profit per trade from your strategy is large enough that paying the spread doesn't significantly impact upon it, then all of this probably isn't worth bothering too much about. If you look back to post 50 you'll see that my starting point was a strategy where the average profit was about $10 per trade. Pay the spread in and out on this in the ES and you're out by $15 per trade. Use limits only and you might miss too many winners and also lose money. At this point execution becomes the difference between a great strategy and a terrible one. If you were swing trading from a daily charge and averaging a profit of $500 per trade, would you be too bothered about paying the spread? I wouldn't. BlueHorseshoe

Hi Predictor, Thanks for posting. I think the description above describes pretty much the same conclusion that I had arrived at, which is encouraging. What I am now doing is considering ways to go further than this, which obviously means working with estimations rather than certainties. I was wondering if you had any insights about how to do this? For instance, ways to estimate what proportion of cancelled orders are behind or ahead in the queue? Cheers, BlueHorseshoe