BlueHorseshoe

I see your point now. In terms of my description they fit the term I (incorrectly) used of market orders, in that they cross the spread to produce volume at the other side. But they're obviously limits. My head hurts! BlueHorseshoe

Hi Horace, Just to be clear - I can't offer you any advice at all about what you should do with any of this information as I don't daytrade and have never applied any of it. If and how you incorporate information about liquidity into your strategy is entirely up to you. Rather, I was just offering an answer to your original question about how price can move counter to bid:ask volume ratio. Personally, I wouldn't be inclined to look at this information on the DOM. I would get an algorithm to look at it - it will be a lot quicker, won't get tired, bored, or emotional, and won't make clumsy mistakes. Such an algorithm might have four inputs: bidsize, asksize, volume at bid, volume at ask . . . depending on what you wanted to achieve. Others on here such as MightyMouse probably wouldn't agree with that last paragraph, as the appraoch a trader takes will be based on that individual's ability to process information without relying on computers etc. Hope that helps. BlueHorseshoe

Hi SIUYA, I have no wish to make anything more complicated than it needs to be, and I agree with you that a guestimate is both all that is possible and all that is required. I'm not sure that was what Paul was saying though - I understood him to be saying that a guestimate was pretty much useless. I certainly agree that a guestimate will often be useless, but I think what is fascinating is that it is possible to know whether or not the guestimate is useless. In his example a guestimate could be made, but clearly it was of little use - such a guesstimate could be thrown out the door (read: order cancelled), and the process repeated until a guestimate in which one had greater confidence was achieved. Surely the option to shrug of anything that isn't high probability and simply move on to look for the next decent opportunity is the main advantage of daytrading? - one opportunity is always hot on the feet of the last. BlueHorseshoe

Hi SIUYA, Thanks for the correction. As far as the ES/CME is concerned though, aren't all orders essentially limit or market? I thought FOK etc were just simulated within a platform, and then implemented in terms of one of these basic order types at the exchange? Cheers, BlueHorseshoe

Er yes, just a bit!!! Thanks - I think I need to do a bit more investigation into exactly what data is widely disseminated by the exchanges and which platforms offer it. This information would make it possible to calculate things like average order size. One could also 'remember' information from before an actual order is placed so that if, say, depth increased by 389 as orders increased by one (implying a single order of size 389), then one places one's own order, then depth increases behind this, and then depth decreases by 389 at the same time as orders decrease by one, it could be reasonably assumed that the 389 order ahead of one's own in the queue has been cancelled, rather than a total of 389 one-lots, some of which were ahead and some of which were behind . . . BlueHorseshoe

Despite my sarcasm I'm no programmer - you only have to take a look at the number of stupid questions I ask over in the easylanguage forum to see that Not that I'm saying no to the $440 mil however . . . You don't know anyone in HR at Knight by any chance, do you? :rofl: BlueHorseshoe

I received a PM from someone asking about bid and ask volume, and I thought the reply might be useful to other people, so I'm taking the slightly egotistical step of re-printing it here: Please feel free to correct any glaring errors - I'm no expert! BlueHorseshoe

Hi Paul, I do agree with what you're saying - there is no way that I (or anyone else) could know without having more data. So beyond what I described in that second post it would indeed be a guess/estimate. I'm not sure why you think that the formula need be arbitrary, however, or not prove sufficiently informative to be useful? An SMA, although it seldom corresponds with actual price, can provide a workable estimate of the underlying trend state of a market. And some estimates are better than others. If a method of estimating PIQ is only sufficiently accurate to improve a strategy by a small amount, but is unique in its ability to improve this aspect, then I would consider it worthwhile. But just to be absolutely clear: I'm not suggesting that it is possible to arrive at a highly accurate estimate of PIQ, and certainly not one that is known with certainty. Glad to hear working for uk banks improved your spelling, even if it tarnished your soul - maybe you should check your contract with the devil - it mightn't be valid in your jurisdiction if the terms are poorly spelt - or should that be spelled? BlueHorseshoe

Hi MightyMouse, Thanks for the very helpful reply. I had been puzzling over this myself. I used the Infinity platform in sim for a few months last year, and I was pretty certain that this also estimated fills in some manner (they certainly didn't happen as soon as price was touched). I guessed that the algorithm may have been something as simple as filling an order once a certain number of contracts traded at that price, or assuming that the order occupies a position at a fixed fraction from the front of the queue at the time price is touched, and then counting in the traded volume from there. Given that these firms must actually be routing a shed load of real orders, couldn't they just marry a sim order to the last placed real order, and 'fill' the sim when confirmation of execution of the actual order was received? This makes me wonder about TradeStation, which I have never used in sim, but tends to be pretty transparent in how it operates if one knows where to look - a good question to ask Onesmith or Tams, I reckon. And it couldn't hurt to ask the question of a few other firms as well. Cheers for the suggestion! BlueHorseshoe

A further thought on PIQ: Given that an increase of n in depth could correspond to either the addition of n orders or the addition of 2n orders and simultaneous cancellation of n orders etc . . . does change in depth provide a way to measure decay in confidence for an estimation of PIQ? Mind you, if depth remained completely static then orders could still be being added and pulled in equilibrium, in which case confidence would decrease with time. Sorry, thinking aloud there . . . BlueHorseshoe

Hi Paul, Thanks for the reply. As far as my second post on PIQ is concerned, if the number of contracts bid/offered never falls below the number at the time of joining the queue - 5000 in your example - then PIQ could indeed be anywhere between 1 and 5001, exactly as you state. Given the constraints of your example then this, as far as I can work out, is as close to certainty as anyone can get. I'm not sure that your example is particularly representative though - at a guess most new orders and cancellations are for one-lots or thereabouts, not 5000 lot elephants. Is it necessarily any more likely that the next change to depth will increase it, rather than decrease it? Play your example the other way around: there are 5000 queued, I add 100, 5000 are cancelled, 5000 are added: my PIQ is now 1-100. I can know this because the cancellation occurred first. Or, given the relative granularity of order size, might it be that cancellations are distributed with equal probability throughout the queue, so that if a queue falls from 100 to 90 it is probable that where my PIQ was 50 it is now 45? I know that's a very unlikely simplification (at a guess, a cancellation is many more times likely to occur near the back of the queue than the front) but you can see where this could lead with further development. Of course what I, you, or anyone else did or didn't do with a rough estimate would be totally dependent on broader strategy. But, even taking your slightly extreme example, here are some possible responses depending on a trader's goals/motives/circumstances: 1. 5100 orders queued, PIQ 0<>5001, not certain enough - cancel order. 2. 30000 orders queued, PIQ 0<>5001, then see if order fills - plenty of liquidity behind to flip the position back at market and scratch the trade. 3. 5100 orders queued, PIQ 0<>5001, order flow poor and need to establish position - cancel limit and pay spread. 4. 5100 orders queued, PIQ 0<>5001, order flow strong and known PIQ at next price tier good, cancel order and wait for fill at next price. Broadly speaking, this will depend on whether this information was being used merely to aid execution where liquidity is incidental to a strategic goal (eg point 3), or whether gaming liquidity was in itself a strategic goal (eg point 2). Working out an estimate of whether reduced depth is due to cancellations from the front of the queue or later arrivals is certainly a challenge, and there's a very real possibility that I won't be able to arrive at even a very crude way of estimating this. The purpose of the thread was to try and improve my chances by inviting input from people like . . . well, like yourself. By the way, are you from the UK - I just noticed your spelling of 'cancelled'? Cheers, BlueHorseshoe

For anyone who has followed my previous posts on Position In Queue, I now want to try and extend this with a model to estimate position inside of the bounds set out in the second post. So far my only real idea is to model the relationship between known market depth and another known output (price?) as a hidden markov process. Does anybody have any other suggestions? Can anyone think of any way that something like a concept of time decay could be employed? I'll also try and provide a couple of better examples of the approach used in the second post, using real bid and ask data rather than a proxy. Thanks, BlueHorseshoe

Well, that's a fair point. As you note, trading is primarily about risk and money management, so the crystal ball doesn't even need to be terribly good. Just good enough. Ha! You should know! [note to mods: please can we get a 'flaming devil' smiley?] BlueHorseshoe

Really? As a programmer I imagine you don't need me to patronise you with examples of the sort of things such firms exploit, so if that is the conclusion you've reached then I'm not sure what to say. Maybe you're right, who knows? I think twenty years ago I was maybe just learning to program in BBC BASIC - what sort of software development did you do? Regards, BlueHorseshoe

I don't know the answer to these questions and I'm not sure anyone else does - this would tend to support my point that there is no clear answer and that it's 'shades of grey'. If my algorithm enables me to front run in exactly that same way as a firm with flash orders, does some Joe Public investor (it's always 'a granny from Milwaukee' in the books ) really give a damn about the difference? The end result is the same, afterall . . . BlueHorseshoe

I'm not sure how accurately it is possible to gauge the reaction of price to volume further out, but at the time the orders are matched, prices trade at bid whenever a sell market is matched with a buy limit, and at ask whenever a buy market is matched with a sell limit. What you're (possibly) not seeing if you are just looking at volume at bid and offer is the depth of the book - the number of buy orders traded at the ask might be very large compared to the number of sell orders traded at bid, but if the number of sell limits is enormous and keeps getting refreshed, the large number of buy market orders will not push price through those limits. Only once all the limits are exhausted can price trade at the next tier. In other words, comparing volume at bid with volume at ask might not be that informative; try comparing volume at ask with depth of book at ask, and volume at bid with depth of book at bid. Hope that helps - if it doesn't make sense then let me know! BlueHorseshoe

Dear Knight Capital, I wrote the following algorithm to assist you in your recovery and return to profitability: For programmer=some to all Begin If programmer<=dogshit then donotemploy=true; End; Hope it helps, BlueHorseshoe ps. there's no need to keep it secret - everybody else already knows!

Predictor, Two tediously long explanatory posts later (#50 and #64), and now absolutely anybody reading should understand exactly what you meant in the post above!!! It would be great to hear any further thoughts you have on this, as obviously an experienced tape-reader is going to come at it from a slightly different angle to the mechanical approach I gave. Errors, shortcuts, hints, tips, cheats - please feel free to fire away! Cheers, BlueHorseshoe

Hi Paul, I meant exactly the same as what you outline above - maybe I was just unclear. There's obviously a world of difference between the certainties of a flash order and the estimations of any algorithm. However, I'm not sure that this means that an algorithmic estimation shouldn't qualify as frontrunning. This rests on the definition of "information". Front-running is acting in advance on information that is not available to the market at large (such as the broker call in my original example, or indeed a flash order). My question is this: is acting on information that is not available to the public due to the complexity of its derivation, from data that is available to the public, to all intents and purposes the same as front running? If you're capable of re-constructing and 'towing' an iceberg (without the help of flash orders), could the information you're basing your decisions upon really be described as "public"? BlueHorseshoe

"No one’s benefitting from the fact that the entire market could blow up at any second." Nassim Nicholas Taleb might be! There was a great article on Taleb and Niederhoffer in the New Yorker (its been posted on TL before) . . . bear with me . . . here we go: gladwell dot com - blowing up

The first chart image attached below focuses on thirty minutes of trading earlier today. I have marked the close of a price bar at 09.07 with a green arrow. Now, suppose at this time you (or your strategy, or the strategy I showed backtests for yesterday) decide that (for whatever hypothetical reason) you would like to buy a pullback to 1403. So, at 09.07 you place a buy limit order at 1403, marked with a dotted green line. The second image I have attached is a 10-tick chart. I have marked your limit order with a dotted green line, and the time of entry with a green arrow, the same as on the first chart. As you can see from the subsequent price action, you’re at your very best today, and you have predicted the turning point where the market reverses and resumes its previous uptrend to the tick (must be those magic trend-lines you drew!). Now, was your limit order filled? Where were you in the order queue? There are circumstances where you can know that you definitely were filled, but after that it becomes an estimation game based on mathematical modelling of probable outcomes. Here is how you can know when you were definitely filled: At the time your order is placed, a time we will call T1, we consult the order book and note the number of participants already bidding to buy, a number we shall call B, at the same price level of 1403. We’ll call this number X. X=B at T1, and in the case of our example X=14. On the upper sub-pane of the second chart, X is represented by a white line and B is represented by the blue histogram. You can see that at the time the order is placed B and X are the same; after that B changes as the number of bids change, but X remains the same. X is the number of people ahead of you in the queue at the time T1. Assuming their orders remain in place until such time as the market reaches this price level, a time we shall call T2, X sellers will have to jump the spread with market orders to be matched with those of the X buyers ahead of you in the queue, before your order can be filled. If at any time between T1 and T2 the number of participants bidding to buy, or B, should fall below X, then this indicates that the number of orders ahead of yours in the queue has decreased by X-B. Continuing in this vein, we might also think that if B increases above X, as it does several time when the blue histogram bars are greater than the white line, then B-X would give the number of orders in the queue that are behind yours. The reason this is not a valid assumption is that those ahead of you in the queue may cancel their orders at the same time as new orders of greater quantity may be introduced to the back of the queue. Increasing the granularity of your study may reduce some of the error here, but it won’t eliminate it. However, should B increase above X, then B-X will give you a value for the minimum number of orders that are behind you in the queue. The number may be larger, if cancellations have moved you forward in the queue, but it can never be smaller because no order can jump ahead of you in the queue. One scenario that may already have occurred to you is a sequential concatenation of the two boundary definition events we have described above. Suppose that at a time between T1 and T2, that we can call Tj, B=X-100. At a subsequent time, Ti, B=X+200. If at Tj Y=X-B, then at Ti (B-X)+Y=R, where R is the number of orders behind us in the queue. These events need not be sequential, of course, and can be far more simply considered by using another storage variable . . . We can generalise all of this by keeping track of the lowest value assumed by X-B after T1 using a variable K, which is initialised at a value equal to B at T1, and thereafter provides an estimation of our position in the order queue, Z. Z is represented in the upper sub-pane by the red histogram bars – notice that though it can decrease with B, it never increases. We can state this something like: Thus we can calculate a maximum value for Z and a minimum value for R. min(Z)=K max®=B-K Possible utilities of R would make a great topic for a different post. Returning to our initial question and examining the number of orders that were matched at the limit price, shown by downticks in the lower sub-pane, yielding a total of (10+60+17+11)=98 buy limits filled at 1403, we can see from the value of the red histogram bar that the highest position you might have occupied in the order queue is 12. Thus we can conclude with certainty that your order would indeed have been filled (as would another 86 orders queued behind ours). This primitive model has now defined some boundaries of certitude. We can move inside these boundaries by modelling order flow, but that’s a topic for another post! BlueHorseshoe ps. in the example charts above, for reasons of clarity when handling a live data feed, I have substituted a derivative of price for bidsize. pps. apologies for the scrappy little gif formula - there's no symbol font for me to do this in the forum.

My understanding was that it had been banned - or am I mis-remembering? Maybe I'm thinking of quote stuffing . . . Also, I thought flash feeds were only an option with certain exchanges (mostly equity) - I didn't think that the CME, for instance, had ever offered any kind of flash feed. If anyone knows any more about this then please speak up! Thanks, BlueHorseshoe

Hi Paul, HFT is a subclass of Algorithmic trading, and much algorithmic trading is totally benign, I agree. Flash orders are a slightly different issue, surely - a more literal modern variant of frontrunning than my iceberg example? Without any flash privelleges a firm with sufficient algorithmic sophistication would be capable of, in my example, identifying and 'towing' an iceberg in real time. A flash order is a bit like you being able to see tomorrow's closing price today. The type of front-running' that I was describing is more like me being able to predict tomorrow's closing price with great confidence because I can do a bit of maths with today's closing price that most people can't. If you're reading this and want privelleged access to flash orders then approach an exchange and offer to pay them whatever fee it is they require. If you can afford literal frontrunning/flash privelleges then don't bother with whatever pauper's version I am able to suggest here. If someone reading this wants to assess order flow / determine their position in queue / reconstruct icebergs / etc in real time and without access to flash orders, then this thread is a place to find out how . . . But only if I'm clever enough to work out how!!! BlueHorseshoe

I am sure that there are instances where HFTs, like any other firm, commit outright fraud or clearly violate exchange rules. In general though, I think this is very much shades of grey. Consider a scenario twenty years ago: My broker friend receives a large buy order from an institutional customer. He picks up the phone and lets me know right away, and I buy some before he begins to execute the large order. As the large order begins to move the market a few ticks, I flip these contracts/shares back to the institution, who is more than will to snap them up, leaving me with a nice little scalp. This is clearly front-running, and is illegal. Now consider the following scenario, probably occuring somewhere in the bowels of Aurora as I write: An institution places a large buy order. Wary of HFTs and the fact that they have a lot to buy, they try to conceal what they're doing . . . They use a nifty little iceberg algorithm which slices the order up into chunks that reload following a random time lag plus some other secret criteria. Nice try! Inside your liquid-nitrogen cooled colocated computers, your electronic eyes are scanning the market, picking through the data - exactly the same data that every other market participant can see. Your algorithm does a bit of number crunching, and clocks the presence of the institutional buy order. Nobody else spots it. There's no call from a corrupt broker, there's no dinner-party-tip-off from the CEO; your algorithm sees it plain and clear in the numbers - as good as any phone call or inside tip - you start buying ahead of the iceberg . . . (I believe it's known as 'towing' the iceberg) You're acting on data available to all, but on information available only to you. Is that front-running? Is it legal? Is it fair? Please note that the principal advantage in the second example is not speed. It's your algorithm, which is able to derive information that other market participants are not. The fact that another firm isn't smart enough to figure this out is a bit like the broker in the first example calling said firm and the phone being answered by a two year old child. It's not really your fault if everyone else is a bit stupid, is it? For virtually anyone reading (myself included!), the reason that you don't do this isn't because you only trade counter-trend, or because you operate in a higher timeframe, or because you base all your trades on candlestick signals . . . It's because you don't have that algorithm. To claim otherwise is a bit like saying that the reason you don't walk on water is because you prefer walking on carpets. In my first post I said "if you don't know the math it might as well be magic". Perhaps "if you don't know the math it might as well be front-running" is more accurate? Thanks for reading. BlueHorseshoe

Hi Nicolas, I only just checked back on this thread and saw your response - very helpful, thanks! BlueHorseshoe