The Question of Randomness

[russellhq 10]

Hi TN, I generated it in much the same way you did (you will see how in the spreadsheet attached).

 

Basically, I generated a list of 144 rand ticks that went either up, down or no change and added the result to the previous result. After 144 ticks, i started a new "day" using previous days close as the starting point and ran another 144 ticks. I repeated this 200 times to get 200 "days".

 

Then, for each day I took the start, finish, max and min to generate the candle stick chart.

[Do Or Die 10]

Patterns appear in random data, but they are 'random'

 

In the candlestick chart attached few posts back; between bars 126-131 you see bars suddenly go 1/4th of their size. In reality the volatility does not changes so abruptly... it undergoes cycles of expansion and contraction.

 

In the second candlestick chart in this post: http://www.traderslaboratory.com/forums/technical-analysis/10728-question-randomness.html#post127386 you can see a reverse-hammer,hammer,reverse-hammer sequence which I cannot recall on a real chart (between bars 1-16).

[TheNegotiator 13]

  Do Or Die said:

Patterns appear in random data, but they are 'random' ;

 

Er, yeah of course. But the fact they can appear to be similar at all must make you question how you look at charts and the level of importance you choose to assign to patterns.

[Do Or Die 10]

  TheNegotiator said:

Er, yeah of course. But the fact they can appear to be similar at all must make you question how you look at charts and the level of importance you choose to assign to patterns.

 

But again think of it.... who really trades on just a double top.

 

Even those who trade on chart patterns support their decision by a number of discretionary factors: candlestick shapes, volumes, trend, time frame confluence and so on.

 

BTW by 'trader' I mean a 'profitable trader' LOL. Probably the lesson here is for newbies that anything which is too easy does not works.

[TheNegotiator 13]

Lessons are for everyone, not just for 'newbies'. What you take or don't from anything varies based mostly on your experiences. Random data showing aspects of real data is intriguing to me, but I'm not saying that you should take that in any way in particular or even agree that there are similarities. The point is to have a discussion about it and maybe all grow a little wiser.

[Do Or Die 10]

  TheNegotiator said:

The point is to have a discussion about it and maybe all grow a little wiser.

 

Totally agree.

 

BTW I prepared excel sheet which shows RSI Divergence in random data several years ago. I see a discussion going here: http://www.traderslaboratory.com/forums/beginners-forum/10734-divergence-trading.html#post127462 which so far adds no value against the divergence found in random data (i.e. is no positive expectancy, no edge). Maybe I will add that spreadsheet in this thread once divergence has been sufficiently discussed on real data.

 

The most important thing I learned from studying random charts was- If anything looks too easy, it does not works; if anything can be written down in one-liner rule, it does not works.

[RichardTodd 10]

  TheNegotiator said:

Er, yeah of course. But the fact they can appear to be similar at all must make you question how you look at charts and the level of importance you choose to assign to patterns.

 

No. If someone that looks like Pavarotti can't sing, that doesn't make me wonder if Pavarotti was a fraud. The value of random data is that it looks like real data, NOT that it means anything about real data it happens to resemble.

 

(A funny example of a research paper generator that actually got a fake paper accepted for a conference a few years back: SCIgen - An Automatic CS Paper Generator. We shouldn't suspect that real papers on that same topic are all meaningless, just because a random process can create something similar.)

[TheNegotiator 13]

  RichardTodd said:

No. If someone that looks like Pavarotti can't sing, that doesn't make me wonder if Pavarotti was a fraud. The value of random data is that it looks like real data, NOT that it means anything about real data it happens to resemble.

 

(A funny example of a research paper generator that actually got a fake paper accepted for a conference a few years back: SCIgen - An Automatic CS Paper Generator. We shouldn't suspect that real papers on that same topic are all meaningless, just because a random process can create something similar.)

 

Wow, I think that many people are seriously missing the point here.

[Moyyim 10]

  BlowFish said:

Many years ago there was a website you go to and take a test. It simple presented a series of charts (price only) and asked you to select whether it was real or random. (A cursory look and I could not find it today). I remember when I took it there where only a couple of charts that threw me. Real price charts have 'characteristics' that seem pretty rare in random data.

 

Here's one for those who wish to "play" .....

 

ARORA

[supernormal 10]

How was the random data generated? Do you use 1, 0 and -1 to represent the three possibilities: up, unchanged and down? Does each possibility have 1/3 of chance to appear in your entire set of data?

[McKeen 10]

I have to say that this thread is really something special.

 

One can say what they want but the sheer reality really scares me...

 

Perhaps it is time to accept the "fact" stated decades ago; ie, technical analysis is not a science nor a winning concept. :thumbs down:

 

Btw, thanks for that arora-link - it was awesome.

[supernormal 10]

I also have another question besides the ones from the previous post, let's say each possibility (up, down, unchanged) has 1/3 of chance to appear in your entire set of data. How do make the sequence of their appearance random?

 

If it's not 1/3, it's not random. You could continuously place bet on the one possibility with higher odds.

 

If it's 1/3, after a sequence of appearance of a single possibility, the chance of appearance for other possibilities increases.You could then place bet on other possibilities. Doesn't that make randomness "predictable"?

[barrp 10]

It's important not to conflate random and and unpredictable. Yes, the market is random, but that doesn't mean it's unpredictable.

[Peterthemonkey 10]

I think a definition of random is in order.

 

Is it the occurrence of an event without cause? Devoid of any causal factor.

 

All we can do here is guess.

 

Is it the occurrence of an event whose causal factors we ignore completely or partially?

 

Here we may start assigning degrees of probabilities based on our understanding of cause and effect.

[pmwhite 10]

  McKeen said:

I have to say that this thread is really something special.

 

One can say what they want but the sheer reality really scares me...

 

Perhaps it is time to accept the "fact" stated decades ago; ie, technical analysis is not a science nor a winning concept. :thumbs down:

 

Btw, thanks for that arora-link - it was awesome.

 

After getting 19 out of 20 correct on the 2nd try at Aurora, I'd have to say that the human eye can learn to identify the difference between a real data series and a computer generated series, and do it very quickly. But is this skill useful for profit?

 

To the point regarding technical analysis: Most people who come into trading never question the roots of technical analysis or the nature of financial (price) data.That usually comes later after finding that the past may not do a very reliable job of predicting the future, despite how many lines or types of analysis are applied. Traders should be required to take a course in econometrics to learn about how to identify the true nature of price data and how to handle nonstationary time series. Financial data has stochastic trend. Because price is not deterministic, all those fancy TA indicators are useless when applied to raw price data, as they are all designed to work with stationary data.

 

The real kicker is that in most cases those moving averages that are so often applied to price by traders to predict future price movement have a lower order of integration than the underlying price data! It's like trying to predict a complex system with a simple system. Exactly how is that going to work? If only traders could profit from buying or selling a moving average (rather than the underlying price). Then it would be relatively easy to profit.

[Tams 20]

  pmwhite said:

...

 

The real kicker is that in most cases those moving averages that are so often applied to price by traders to predict future price movement have a lower order of integration than the underlying price data! It's like trying to predict a complex system with a simple system. Exactly how is that going to work? If only traders could profit from buying or selling a moving average (rather than the underlying price). Then it would be relatively easy to profit.

 

I know someone who is doing exactly that... trade on MA.

of course he is not simply buy/sell on a MA cross over, there are more to it than that.

but the basic premises of his system is MA, and he is doing quite well.

Never say never when it comes to trading. There are rocket scientists out there who can dream of things we can't even fathom.

[SIUYA 11]

[quote=Tams;127558

Never say never when it comes to trading. There are rocket scientists out there who can dream of things we can't even fathom.

 

very true...just as there are simpletons that can trade as they dont need to over complicate things

 

On the point of randomness and trading.... IMHO this is where the real test for being able to run profits and cut losses can be vital, as even randomly generated numbers can appear to trend.

[cjforex 10]

I forget where I was recently reading, that to the human eye, we expect randomness to look like a relatively even distribution, sort of like white noise. Whereas true randomness does have periods of predicted trendiness (to our eyes).

 

Sometimes even expected, prolonged trends.

 

That being said there were recent extensive academic papers and proofs showing the effectiveness of various technical analysis methodologies.

 

If anyone cares I can dig them up.

 

-cjforex-

[TheNegotiator 13]

I'd like to try to illustrate the idea in a different way. By looking at an event which most would label as random. The coin toss. The idea is obviously that a coin toss produces over many samples, an equal number (or near) of heads and tails based on a 50:50 probability of either results randomly appearing. But is this right? The results do exhibit what we expect to see, but is it on a random basis that each individual result is formed? What about side which the coin is flipped from, strength which the coin is flipped with, atmospheric variables? What would the results be if a mechanical device was used to flip the coin which could use precise force and it was done on the same side of the coin each time and in a vacuum? Well I would suggest the results may not vary in quite the same way. Obviously there are other variables and factors which would be needed to be accounted for in the design of any such experiment, but I think I've made the point.

 

So if something which we perceive to be random is not, how sure can we be each time we make a call and place a trade that the basis for our trade is complete and sound and has no random nature? My feeling is that nothing is random truely, like was indicated in a post earlier in this thread. Random is a creation assigned to events which we don't fully understand and so can't predict other than on a basis of probabilities from a set of previous results.

 

Because we can never be sure of why markets are currently exhibiting certain behaviour, whilst the behaviours themselves are not random, our interpretation of them can only have a probabilty of being correct and therefore show a degree of random nature. As Siuya in different words so rightly pointed out, any strategy must be correctly managed in terms of size, risk:reward etc., etc. and so we can balance out the perceived randomness by being better traders.

 

To reiterate one last time, I don't think markets behave randomly.

[Do Or Die 10]

  cjforex said:

That being said there were recent extensive academic papers and proofs showing the effectiveness of various technical analysis methodologies.

 

If anyone cares I can dig them up.

 

-cjforex-

 

http://www.traderslaboratory.com/forums/technical-analysis/10720-relative-strength-resources-glossary.html#post127409

[RichardTodd 10]

  TheNegotiator said:

I'd like to try to illustrate the idea in a different way. By looking at an event which most would label as random.

 

Ah, ok, now this I agree with. You are right, I could not tell what you were getting at by your initial posts.

 

In mathematics, random variables are introduced into a model in order to gloss over things that are either prohibitively expensive to track, or unknowable. The randomness is in the model, not the reality being modeled. So I agree with you 100%. The coin-flip example you used is actually the one used in more than one introductory book on the subject! You could give better than 50/50 odds on a coin flip if you knew the weight distribution of the coin, the air pressure, the position and velocity of the flipper's thumb at all times, etc. But that's hard, so you tend to go with a simpler 50/50 model that does an ok job of looking like a typical distribution over time.

 

If we were traders making directional bets on coin flips, it would be our job to go to the trouble of measuring the air pressure and watching the tension on the thumb, to get a better idea of which way the coin is going to land. The fact that a 50/50 model can make a reasonable picture of actual coin-flip trials doesn't make this any less possible. This is why a random walk picture of stock prices doesn't change the importance I place on chart patterns one bit. (I'm still confused about why you seemed to say otherwise??)

 

I think what confused me (and perhaps others) about your initial posts was when you said things like:

 

  TheNegotiator said:

Would you trade the product in my chart if you knew it were completely random???

 

... which to me implied that (1) you did not know that it's a fact that you cannot beat a random walk with any strategy, and (2) you thought a product could be completely random. So, no one should trade a random product, but thankfully those do not exist

 

I was also thrown off here:

 

  TheNegotiator said:

However, when you look at historical charts, they have the appearance of being randomly generated

 

To say that charts look like models of charts is true enough, but it just sounds funny to me. To my way of thinking, it is like saying Elvis looks like an Elvis impersonator. Price is a fact, and a random-walk model uses a single idea (that prices tend to move in small amounts relative to where they've been) to make a picture that is superficially similar.

 

To the other points in the thread: Can you tell them apart? Yes. There was also university study in the last few years that showed people were able to discern real price charts from randomly-scrambled ones with a little practice. More sophisticated models (for starters, you can vary the volatility) are harder to discern, but the point is that no matter how realistic the picture is, it's just a model, and does not imply that the real deal is in some way "random" or wholly "unpredictable."

[RichardTodd 10]

  supernormal said:

If it's 1/3, after a sequence of appearance of a single possibility, the chance of appearance for other possibilities increases.You could then place bet on other possibilities. Doesn't that make randomness "predictable"?

 

This is the Gambler's fallacy - Wikipedia, the free encyclopedia. The chance for the other possibilities does not increase in a model built on independent events.

Edited September 9, 2011 by RichardTodd

[Avarice 10]

It is empirically impossible to prove that something is behaving in a "random" way. For something to be perfectly random it has to lack inherent order - so trying to prove market randomness equals trying to prove that it lacks underlying order. Proving such a negative proposition is impossible - it's like trying to prove that there are no cakes in space. While this might be highly probable, providing mathematical proof of said proposition is a different pair of shoes.

 

I remember reading a passage in some book (I think it was Schwager) that a trainee was sitting next to his boss in some firm, looking at commodity quotes. After a couple of hours the trainee asked his boss how he managed to make so much money out of random price movements. The boss picked up the phone, told his trading floor to short the commodity pretty heavily - 30 seconds later prices printed lower. "If one man with a telephone", the boss answered his trainee, "can move the markets with a single phone call, how can it be random?" (I think this was Dennis, not sure tho...)

 

In any case. As always SIUYA has the right of it, randomness or not: it's trade management that makes or breaks your account. To answer your question would I trade a truly random stream of data? Sure, I would. Would I recommend investing in such an instrument? Hell no.

[Do Or Die 10]

  Avarice said:

In any case. As always SIUYA has the right of it, randomness or not: it's trade management that makes or breaks your account. To answer your question would I trade a truly random stream of data? Sure, I would. Would I recommend investing in such an instrument? Hell no.

 

If this were the case mathematicians would turn out as best traders. You cannot make money UNLESS you are able to identify patterns with positive expectancy (non-random).

[Avarice 10]

  Do Or Die said:

If this were the case mathematicians would turn out as best traders. You cannot make money UNLESS you are able to identify patterns with positive expectancy (non-random).

 

I humbly disagree. Profitability of a trading system is defined by your ability to press winners and to run fast in the face of adversity. Mathematicians (or any scholar, for that matter) make for terrible traders. Most think they have to be right in order to win, hence they fail to bail if the trade turns bad. Yep, wide generalizations FTW :rofl:

 

But we disgress...