Trading with Market Statistics XI. HUP

[BlowFish 10]

I wonder if the fact that "The skewness of a random variable X is sometimes denoted Skew[X]. If Y is the sum of n independent random variables, all with the same distribution as X, then it can be shown that Skew[Y] = Skew[X] / √n." can be exploited to come up with a non iterative approach?

 

I am having trouble understanding this. Doesnt it mean that you can simply calculate the skew for the first element (which will be zero right?) and then extrapolate from that. I don't understand how that formula can be correct.

 

 

 

Its from http://en.wikipedia.org/wiki/Skewness

[Head2k 10]

I wonder if the fact that "The skewness of a random variable X is sometimes denoted Skew[X]. If Y is the sum of n independent random variables, all with the same distribution as X, then it can be shown that Skew[Y] = Skew[X] / √n." can be exploited to come up with a non iterative approach?

 

I am having trouble understanding this. Doesnt it mean that you can simply calculate the skew for the first element (which will be zero right?) and then extrapolate from that. I don't understand how that formula can be correct.

 

 

 

Its from http://en.wikipedia.org/wiki/Skewness

I don't think this shows some non-iterative approach. As I said I am not educated in statistics so I am merely guessing, but I think this random variable X is not a single number but actually a discretionary data set or a continous distribution function. So if you want to determine Skew[X] you need to iterate or integrate, anyway.

[BlowFish 10]

Ondrej,

 

Would you mind posting the bit of code that calculates the Skew? I find it easier to understand algorithms than formulas on the whole.

[Head2k 10]

Ondrej,

 

Would you mind posting the bit of code that calculates the Skew? I find it easier to understand algorithms than formulas on the whole.

I am not sure how familiar you are with AmiBroker Formula Language, but anyway, this is the iterative part of my code. Some parts within the iteration are ommited.

i cycle goes through bars, j cycles through price levels. This iteration computes Volume per price (VUcount), PVP, VWAP, Variance and the 3rd central moment (CentMom3). BoxesInRange is the number of price levels in the Volume Distribution function.

 

for(i = iStart; i < BarCount; i++)

{

SumPjVj = 0;

TotalVolume = TotalVolume + Volume;

if((BoxesInRange < BarCount))

for(j = 0; j < BoxesInRange; j++)

{

BoxPrice = Bot + j * TickSize;

if((Low <= (BoxPrice + TickSize/2)) AND (High >= (BoxPrice - TickSize/2)))

{

VUcount[j] = VUcount[j] + VolUnit;

MaxVUcount = Max(MaxVUcount, VUcount[j]);

}

SumPjVj = SumPjVj + (BoxPrice * VUcount[j]);

}

if((frac(i / CalcInt) == 0) OR (i == BarCount - 1))

{

VWAP = SumPjVj / TotalVolume;

for(j = 0; j < BoxesInRange; j++)

{

BoxPrice = Bot + j * TickSize;

Prob = VUcount[j] / TotalVolume;

if(MaxVUcount == VUcount[j])

PVP = BoxPrice;

Variance = Variance + (Prob * (BoxPrice - VWAP) * (BoxPrice - VWAP));

CentMom3 = CentMom3 + (Prob * ((BoxPrice - VWAP) * (BoxPrice - VWAP) * (BoxPrice - VWAP)));

}

}

}

}

[BlowFish 10]

I don't think this shows some non-iterative approach. As I said I am not educated in statistics so I am merely guessing, but I think this random variable X is not a single number but actually a discretionary data set or a continous distribution function. So if you want to determine Skew[X] you need to iterate or integrate, anyway.

 

The way I read this was if you know the skew of the set of variables you can determine the scew of any particular variable (like a newly arriving bar). I'd wager my stats are weaker than yours though!

 

Thanks for the algorithm by the way. I am not particularly familiar with ambibroker but syntax dosen't look too dificult

 

This might interest you http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance this was what inspired me to look for non iterative approaches to VWAP and it's SD's

[Head2k 10]

The way I read this was if you know the skew of the set of variables you can determine the scew of any particular variable (like a newly arriving bar). I'd wager my stats are weaker than yours though!

 

Thanks for the algorithm by the way. I am not particularly familiar with ambibroker but syntax dosen't look too dificult

 

This might interest you http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance this was what inspired me to look for non iterative approaches to VWAP and it's SD's

The way I read this is that if you know the skew of a variable X (variable meant as a function), you can somehow calculate the skew of a varaible Y (again meant as a function). But I don't understand why. Reading Wiki I was also a bit confused, then I realized that in statistics they often use whole functions as variables. I noticed that if a variable is a function it is usually marked with a capital letter, such as X or Y. If a variable is a number (a value of the function at some point) it is usually marked with a small letter, such as x or y.

Thanks for the link, but my indicator for AmiBroker works fast enough to be used in real time on 15sec database for 1 day. I am a newbie and I need to learn a lot so my priorities lie elsewhere. By the way, that condition in the code "if ( frac( i / CalcInt == 0) ... ) sets an interval of calculation (frac is fractional part). When I recalculate VWAP and everything only once per four 15sec intervals, the speed of the indicator almost doubles.

[BlowFish 10]

Slowly trying to get my tiny mind round this. Do you Divide this value CentMom3 by SD^3? I see you add it to VWAP and multiply by the SD for plotting.

 

CentMom3 = CentMom3 + (Prob * ((BoxPrice - VWAP) * (BoxPrice - VWAP) * (BoxPrice - VWAP)));

 

Seems like you should to me. I finally had one moment of clarity (if you pardon the pun) still lots that's hazy. Anyone know of a better place than Wiki to properly learn this stuff?

[Head2k 10]

Slowly trying to get my tiny mind round this. Do you Divide this value CentMom3 by SD^3? I see you add it to VWAP and multiply by the SD for plotting.

 

CentMom3 = CentMom3 + (Prob * ((BoxPrice - VWAP) * (BoxPrice - VWAP) * (BoxPrice - VWAP)));

 

Seems like you should to me. I finally had one moment of clarity (if you pardon the pun) still lots that's hazy. Anyone know of a better place than Wiki to properly learn this stuff?

Yes I do sqrt(Variance) to calculate SD and I do CentMom3 / SD^3 to calculate Skew(3CM). But these operations I do not do within the iteration loop, because AmiBroker AFL is primarily an array processing language. So I do these computations on arrays outside of the iterative loop I posted.

[BlowFish 10]

Yes I do sqrt(Variance) to calculate SD and I do CentMom3 / SD^3 to calculate Skew(3CM). But these operations I do not do within the iteration loop, because AmiBroker AFL is primarily an array processing language. So I do these computations on arrays outside of the iterative loop I posted.

 

Gotcha, thanks. From what I hear AmiBrokers script language is pretty efficient in execution so I guess thats a plus too.

[Head2k 10]

Gotcha, thanks. From what I hear AmiBrokers script language is pretty efficient in execution so I guess thats a plus too.

Today I improved my code so when it updates VUcount (volume per price) the j-loop doesnt go through all prices seeking where is the actual bar, but first I find the bar borders and then the loop goes only within the bar. With this improvement I can compute the indicator on sevreal days of data without any noticable slowdown, so now it is fully usable for real time even for longer periods of high detailed data.

I dont know if AFL is slow or fast compared to other softwares, because I am a beginner and AmiBroker is the first charting software I use But I find it very flexible, so far I could program everything I wanted there.

[BlowFish 10]

Well it sounds like you are doing great with it. Good job! Funnily enough that was exactly the improvement that I made to DBtina's code for PVP. Getting stuff out of loops (or eliminating them altogether) is one of the arts of efficient code. The code I have for VWAP SD and PVP does not have a single loop in it (except to initialise arrays to zero on bar 0).

 

I am now pretty certain that this can be done in a non-iterative manner its just a bit beyond me from the maths/stats end of things.

[BlowFish 10]

Well I have made some progress by completely changing my variance algorithm and basing the 3rd central moment stuff on that. Ondrej could you post the odd chart with the whole days action squashed up? I guess really I am going to have to do a long hand algorithm to test against.

[Head2k 10]

In this post of mine

http://www.traderslaboratory.com/forums/6/trading-with-market-statistics-xi-hup-2735-7.html#post43133

there is the Skew Line displayed for ES 9:30 - 16:15 in the first two of the three attachments. Filenames show dates.

If you need more, just give me a symbol, date and times (from to) and I will make a screenshot. The date should be within the last two months.

 

However, I observed the Skew Line a bit more and I recognize that sometimes PVP to VWAP relationship is better for description of the price action. Sure the Skew(3CM) is the real statistical skew, yet one has to think about what the skew is and what it means.

 

Skew is "missing expected data" compared to normal distribution. For the skew(3CM) the base of the distribution is formed with whole price range for the examined period (e.g. daily range). In this range we expect normal distribution and skew tells us the deviation from normal. Yet one has to ask if the chosen examination interval is relevant. I attached a chart for ES from the last Friday (08/01). Apparently the opening price at 9:30 was quickly rejected and the market never returned to that area again. This action created missing volme in the top of the daily range and thus a skewed distribution. Notice the thin light blue line (the Skew Line = VWAP + Skew(3CM) * SD) wrapped around the 1st SD for the whole day. Looking back one can recognize that setting the start of the examined interval after the opening price rejection would provide a better insight.

 

On the other hand you can notice that the PVP to VWAP relationship described the real situation much better. Thats because skew(PVP) is not so dependent on the daily range, it smooths low volume excesses. It is only as sensitive to daily range as VWAP is.

 

So each Skew tells you a bit different kind of information.

 

EDIT: Also notice the thick violet line showing the daily range middle. Range middle is mostly a good approximation of the skew line.

[BlowFish 10]

Thanks,

 

I do think the PVP presents some problems also, I'll discuss them later. I am really pleased with the new algorithm as it does not seem prone to numerical instability and overflow errors even when cumulating ^3's. Having said that the skew calculation looks like it needs a tweak compared to yours. I enclose my chart for the same day. I'm really not quite sure what needs doing to be honest.

 

Edit: Didn't Pearson suggest 3*(mean-median)/SD? I wonder if VWAP + 3(mean-median) might be useful?

[BlowFish 10]

Ok here is the same chart with the sign flipped. (i.e. Vwap - Skew * SD plotted). Thats looking kinda close. Could you post todays ES if you get a moment.

[Head2k 10]

If your Skew Line is the blue one you have it wrong. I will recap the procedure.

 

Asume you have an i-loop going through all bars since the start of the examination period.

 

For each i you need to do the following:

 

1. You need to determine Volume per price (the volume distribution function). Lets say you will have an array VD, where VD[j] will be volume per j-th price. This VD array will change for every bar (every i).

 

2. You need to go through this array in a j-loop and calculate the following:

 

a) Probability of j-th price: Prob[j] = VD[j] / TotalVolume. TotalVolume is the cumulative volume since the start of the examination period.

 

b) You need to calculate the 3rd central moment for every bar (every i). You again need to do it in a j-loop.

CM3 = sum( (Price[j] - VWAP)^3 ). The sum goes through all j's.

 

c) You calculate the Skew = CM3 / SD^3. This you dont have to do in a j-loop.

 

The j-loops dont have to be separate, you can do everything in two. I use three j-loops for every i. In the first one I determine the VD. I do this in a separate loop because I dont go through all j's but only through those representing the actual bar. To VD array I add volume cumulatively from the start of the examination, so I dont need to recalculate it whole every bar, I need only to add the current bar volume to it.

The second loop I use for VWAP calculation and in this one I need to go through all prices.

The third j-loop I use for Prob and CM3 calculation and in this one I need to go through all prices. For this calculation I must know VWAP, so I cannot integrate the first and second loop. But if your software calculates VWAP for you and you dont need to calculate VWAP yourself, you can use only 2 loops.

[Head2k 10]

The chart for today until present time.

 

EDIT: Btw it was an awful creepy movement today before FOMC, I am glad that after 2hrs of wtching this I decided not to trade at all today.

Edited August 5, 2008 by Head2k

[BlowFish 10]

Thanks for your continued dialogue! Yeah its wrong It is a continuous algorithm though so it doesn't use loops at all. It is based on an algorithm by West the weighted incremental algorithm here http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance. I think its real close now but something is a wee bit out. It will run on months of ES tick data though! Its a pretty neat and not too difficult to understand. I think the logic is pretty good as everything is OK except the skew. My hunch is that its the last bit where I divide (n-1) * sumweights. Not sure why though

 

n = 0

foreach x in the data:

if n=0 then

n = 1

mean = x

S = 0

C=0

sumweight = weight

else

n = n + 1

temp = weight + sumweight

S = S + sumweight*weight*(x-mean)^2 / temp // sum of weighted ^2's

C = C + sumweight*weight*(x-mean)^3 / temp // sum of weighted ^3's

mean = mean + (x-mean)*weight / temp

sumweight = temp

end if

end for

 

Variance = S * n / ((n-1) * sumweight) // if n = 1, omit n/(n-1)

SD = Sqrt(Variance)

Skew = C * n/((n-1) * sumweight) / cube (SD)

[Head2k 10]

I am not a programmer. To programe in AmiBroker I needed to read a manual first. So I am afraid understaning your code would cost me too much effort. Anyway, if C is the sum of weighted ^3's, why don't you calculate the skew just as Skew = C / cube(SD) ?

Maybe I missed something...

[BlowFish 10]

Well another day another algorithm this one based on Terriberry's paper 'Computing Higher Order Moments Online". This one is realy neat but I don't fully understand how to weight it properly.

 

I wonder if you would mind posting todays ES Ondrej? This is what I have but I know (well am pretty sure) the 3CM isn't weighted correctly. Its the magenta line. At least its the right shape now (I think).

 

Cheers,

Nick.

[Head2k 10]

Not so far away from what I have.

 

As a side note, this discussion merely belongs to the Skew thread, not this one. I posted my first post here as a reaction on a post when the 3rd central moment was first mentioned. But now we spent several pages discussing skew in a HUP thread... Maybe some moderator could move it, if it is not too difficult?

[BlowFish 10]

Not so far away from what I have.

 

As a side note, this discussion merely belongs to the Skew thread, not this one. I posted my first post here as a reaction on a post when the 3rd central moment was first mentioned. But now we spent several pages discussing skew in a HUP thread... Maybe some moderator could move it, if it is not too difficult?

 

Yes I was thinking the same thing if a mod wouldn't mind?

[moon 10]

Hi Jerry,

Thanks for the great threads!

 

I was thinking about two things you mentioned in the treads:

1. Risk tolerance vs. Stop-Loss: If I am using risk-tolerance of 2% of account of 100K (i.e. my risk tolerance is 2K) and I am trading a specific market (e.g. YM), then actually the risk tolerance can be translated to 2K/5 = 400 points for every trade.

Do you enter a trade with default SL of 400 points away?

 

2. After examining the VWAP/PVP/SDs and their relationship you decide to enter a trade Long. Your risk tolerance is still far away, but the market conditions changed (e.g. vwap and price are now below PVP) - would you exit the trade? Stay in the game till your risk-tolerance is hit?

 

Thanks,

 

moon.

[jperl 11]

Hi Jerry,

Thanks for the great threads!

 

I was thinking about two things you mentioned in the treads:

1. Risk tolerance vs. Stop-Loss: If I am using risk-tolerance of 2% of account of 100K (i.e. my risk tolerance is 2K) and I am trading a specific market (e.g. YM), then actually the risk tolerance can be translated to 2K/5 = 400 points for every trade.

Do you enter a trade with default SL of 400 points away?

 

2. After examining the VWAP/PVP/SDs and their relationship you decide to enter a trade Long. Your risk tolerance is still far away, but the market conditions changed (e.g. vwap and price are now below PVP) - would you exit the trade? Stay in the game till your risk-tolerance is hit?

 

Thanks,

 

moon.

 

Moon, your calculation is correct for YM. This does not mean that you simply enter a trade without managing it. If, as you point out the price action is such that the trade no longer makes sense to you, then you simply exit it or reverse it. The point about using risk tolerance, is that you are not locked in to a course of action which takes you out of the market too early because a stop was hit. Rather, it gives you the opportunity to adjust the trade by either adding size to it or reversing it without the fear that a pre-set stoploss creates.

[moon 10]

Hi Jerry

Thanks a lot for your answer.

 

Main reason for my previous question comes from what you wrote few pages back at this thread (as an answer to Trader333)

 

“Also think about what you would have done if the trade at the VWAP had not gone your way. Would you have exited the trade at the PVP or done something else?”

 

Trader333 answered:

I would have exited at either the PVP line or 1st Standard Deviation away from entry whichever was nearer at the time.

 

And you wrote:

That is what most traders would do and in my opinion leads to a slow bleed of your capital

 

By this answer I assume you would have stayed in the trade although market conditions changed. (?)

Can you please advise what would you have done?

 

Thanks!

 

moon