The Structure of Non-randomness

[onesmith 10]

I have heard that randomly created synthetic data is able to mimic many features of market data and even be indistinguishable (from real market data). Assuming everyone contributes toward creating the non random structure within the real data ......... What is the markets structure?

 

What is the non randomness that distinguishes real data from random data.

[Seeker 11]

I have heard that randomly created synthetic data is able to mimic many features of market data and even be indistinguishable (from real market data). Assuming everyone contributes toward creating the non random structure within the real data ......... What is the markets structure?

 

What is the non randomness that distinguishes real data from random data.

 

This seems to be a debate on a few threads here.

 

The number of people in a queue at a shop is random. It goes up, it goes down, it doesn't go below zero, I can't say with certainty what this number will be in 5 minutes or indeed any future time, so in that sense it has randomness. So someone just observing the queue will see random numbers changing with very little pattern. If that's what we look for, that's what we find. But how many people's actions are actually random in that queue?

 

Having the appearance of being like a random walk, does not make it a random walk. If we paid attention, couldn't we see when people are more likely or less likely to join the queue? We obtain very little from assuming it's all random, but we might gain something by thinking that it's not, and that each of these people has some intent, so that for example, we may conclude that as closing time approaches, the number of people in the shop will quite likely quickly move to the queue.

 

Experiments have been done on whether humans can tell random data from real financial data, and it was demonstrated that humans could. I quote from a recent paper which researched this:

 

"We find overwhelming statistical evidence (p-values no greater than 0.5%) that subjects can consistently distinguish between the two types of time series, thereby refuting the widespread belief that financial markets “look random”."

 

And these were not necessarily traders taking this test.

Edited April 15, 2013 by Seeker

[addchild 10]

I have heard that randomly created synthetic data is able to mimic many features of market data and even be indistinguishable (from real market data). Assuming everyone contributes toward creating the non random structure within the real data ......... What is the markets structure?

 

What is the non randomness that distinguishes real data from random data.

 

 

From my (limited) experience, random data is largely used to gauge the statistical significance of results.

 

It might also be used for theoretical market participants in building limit order book models.

 

Also, testing an idea against random data to verify results would imply the tester believes markets are not random.

[DbPhoenix 42]

What is the non randomness that distinguishes real data from random data.

 

In terms of making money, it doesn't make the slightest bit of difference.

 

What is the markets structure?

 

The market can go up, down, or sideways (or, for extremely brief periods of time, it can stand still). If one knows how to tell the difference between up, down, and sideways, he can exploit -- i.e., profit from -- these movements. If he doesn't, or can't, then whatever profits he realizes will be purely accidental, if he profits at all.

[natureboy1618 10]

In terms of making money, it doesn't make the slightest bit of difference.

 

That's the real point. Even great trend trading methods can have win rates of 30-40%. Whether the markets are random or not doesn't even matter. The only thing that matters is how much is made on average vs how much is lost on average and the frequency of both over time (and keeping the trade distribution close to that average - no big losses). With a large enough reward to risk, you can make money with 10% accuracy, probably on 'random' price data and real price data. Purely academic..

 

As Steve from Santa Barbara pointed out, randomness itself is hard to model accurately with a computer (so even our estimations of 'random' are not in fact pure randomness).. You can use Brownian motion of bacteria as a random model... even then, you see trends! They call it 'drift'. So even with natural models of randomness you see trends. Now do trends in prices occur more frequently than drifts in brownian motion? I'll leave that to a PhD.. i'm interested in making money. Lets get a thread going about making money.. step 2..

[onesmith 10]

This seems to be a debate on a few threads here.

 

The number of people in a queue at a shop is random. It goes up, it goes down, it doesn't go below zero, I can't say with certainty what this number will be in 5 minutes or indeed any future time, so in that sense it has randomness. So someone just observing the queue will see random numbers changing with very little pattern. If that's what we look for, that's what we find. But how many people's actions are actually random in that queue?

 

Having the appearance of being like a random walk, does not make it a random walk. If we paid attention, couldn't we see when people are more likely or less likely to join the queue? We obtain very little from assuming it's all random, but we might gain something by thinking that it's not, and that each of these people has some intent, so that for example, we may conclude that as closing time approaches, the number of people in the shop will quite likely quickly move to the queue.

 

Experiments have been done on whether humans can tell random data from real financial data, and it was demonstrated that humans could. I quote from a recent paper which researched this:

 

"We find overwhelming statistical evidence (p-values no greater than 0.5%) that subjects can consistently distinguish between the two types of time series, thereby refuting the widespread belief that financial markets “look random”."

 

And these were not necessarily traders taking this test.

 

Other than the players spoofing orders to hide what might otherwise be viable info there shouldn't be many random orders in the queue. Is the queue a viable source of the non random footprint winning and losing traders leave in the market?

 

I'm curious about that study. What characteristics of the real data enabled determining it wasn't random?

[onesmith 10]

From my (limited) experience, random data is largely used to gauge the statistical significance of results.

 

It might also be used for theoretical market participants in building limit order book models.

 

Also, testing an idea against random data to verify results would imply the tester believes markets are not random.

I suspect there's a lot of info to be gleaned from what appears as randomness. I'm particularly interested in randomness from the point of view of determing when and how it might deliberately or randomly conceal otherwise non-random aspects of the data. My focus is the non-random aspects and integrating the effects of randomness such as this afternoons Boston Marathon incident interupting an otherwise logical buying opportunity.

[onesmith 10]

In terms of making money, it doesn't make the slightest bit of difference.

 

The market can go up, down, or sideways (or, for extremely brief periods of time, it can stand still). If one knows how to tell the difference between up, down, and sideways, he can exploit -- i.e., profit from -- these movements. If he doesn't, or can't, then whatever profits he realizes will be purely accidental, if he profits at all.

 

 

From the point of view of differentiating real data from synthetically created random data the utility is limited to what I learn about real data. Within the realm of just the real data it is worthwhile distinguishing between random and non-random. For instance up is not always up. Just because direction appears obvious does not make it the true direction. I have observed losers buying dips and winners capitalizing on that. Correlating the actions of winners to anything that repeats is the essence of trading. Along with understanding the basis (losers leaving non-random footprints) or random lack of a basis for moves that fail.

[Seeker 11]

Other than the players spoofing orders to hide what might otherwise be viable info there shouldn't be many random orders in the queue. Is the queue a viable source of the non random footprint winning and losing traders leave in the market?

 

I'm curious about that study. What characteristics of the real data enabled determining it wasn't random?

 

It's not known what characteristic enabled people to determine which was real and which was random - just that they could. It was a test only to see if humans could detect the difference. I think you can still see what the test was like here:

 

ARORA

 

Random is a tricky word. Anything you do or write is somewhat random to ME. It's not random to you, because you choose it, but it is to everyone else. I say somewhat because you are still using words and sentences, there is still structure and I am familiar with it, so it's not entirely random. So when you're asking if things are random in the market, the question is 'to whom'? And how random? The markets are random to all participants, but they are not entirely random. At least that is how I see it.

[onesmith 10]

That's the real point. Even great trend trading methods can have win rates of 30-40%. Whether the markets are random or not doesn't even matter. The only thing that matters is how much is made on average vs how much is lost on average and the frequency of both over time (and keeping the trade distribution close to that average - no big losses). With a large enough reward to risk, you can make money with 10% accuracy, probably on 'random' price data and real price data. Purely academic..

 

As Steve from Santa Barbara pointed out, randomness itself is hard to model accurately with a computer (so even our estimations of 'random' are not in fact pure randomness).. You can use Brownian motion of bacteria as a random model... even then, you see trends! They call it 'drift'. So even with natural models of randomness you see trends. Now do trends in prices occur more frequently than drifts in brownian motion? I'll leave that to a PhD.. i'm interested in making money. Lets get a thread going about making money.. step 2..

 

Risk/Reward is important but discernible risk/reward removes your entry from the random category. The footprint of your having taken the time to learn to calculate r/r forms part of the non randomness that's the essence of why markets are tradable. This isn't academic. Brownian drift of markets tends toward deterministic models dissimilar to random models. Not all buyers and sellers act with the same (random) level of disregard for stucture. Winners and losers both leave footprints. Order and structure that's non-random and discernable. From that point of view studying THE STRUCTURE OF NON RANDOMNESS cuts to the essence of what it takes to make money.

 

I'm not interested in swaying opinions or convincing anyone of anything. I prefer keeping my beliefs and opinions to myself. I'm interested in hearing everyone elses thoughts and learning something here. Please use this exception to my more normal not sharing as an opportunity to throw your best at me and convince me what matters in your world is the really important thing I should be thinking about. Particularly within the realm of the existence or non existence of the non random structure I am claiming is the basis and driving force behind all markets.

[DbPhoenix 42]

From the point of view of differentiating real data from synthetically created random data the utility is limited to what I learn about real data. Within the realm of just the real data it is worthwhile distinguishing between random and non-random. For instance up is not always up. Just because direction appears obvious does not make it the true direction. I have observed losers buying dips and winners capitalizing on that. Correlating the actions of winners to anything that repeats is the essence of trading. Along with understanding the basis (losers leaving non-random footprints) or random lack of a basis for moves that fail.

 

You're thinking about this too much. Instead of trading what's in front of you, you're wondering whether or not it's real.

 

Just trade it, and you'll do better than most. Or overthink it and risk doing worse.

[SIUYA 11]

all this talk about randomness poped into my head when reading this news/infovertisement

 

What to do when everyone is right

 

.............

[DbPhoenix 42]

all this talk about randomness poped into my head when reading this news/infovertisement

 

What to do when everyone is right

 

.............

....................

Edited April 16, 2013 by DbPhoenix

[BlueHorseshoe 10]

 

I'm not interested in swaying opinions or convincing anyone of anything. I prefer keeping my beliefs and opinions to myself. I'm interested in hearing everyone elses thoughts and learning something here. Please use this exception to my more normal not sharing as an opportunity to throw your best at me and convince me what matters in your world is the really important thing I should be thinking about. Particularly within the realm of the existence or non existence of the non random structure I am claiming is the basis and driving force behind all markets.

 

Not answers, but various thoughts . . .

 

• If you muddle up the data in a time series (monte carlo), then the way that each member of the series relates to all other members is random, but . . . the time series will still have the same outcome, or sum to the same total. The process is random; the outcome is fixed.
   
  
• Suppose you have a known price now, and you think you can predict a future price. Assuming you can, then that still isn't the same as saying that you can predict the path involved in getting there. That would involve a hell of a lot more prediction. Which is one reason that people doing fancy things with trailing stops etc confounds me . . .
   
  
• A lot of the references to 'non-random' structures appear to be rooted purely in price, and how it relates to itself visually within a chart. Is it any wonder if such a limited view of structure should turn out to hide all manner of randomness?
   
  
• What about Mandelbrot - tiny differences in even the simplest and most benign systems can have diverse outcomes over many iterations . . . and a fat-finger error from a one contract retail trader can trigger a market crash . . . Does anyone actually believe this?
   
  
• Talk of footprints - just because an order is not randomly placed doesn't mean that the individual placing it has any kind of directional motive. If someone is operating within a completely different structure of behaviour to yourself, then are their actions really in any worthwhile sense any different to those of someone/thing behaving randomly?
   
  
• Non-random behaviour is predictable if you know the cause. Random behaviour is predictable if you know that there is no cause . . .
  

 

BlueHorseshoe

[onesmith 10]

all this talk about randomness poped into my head when reading this news/infovertisement

 

What to do when everyone is right

 

.............

 

What does your taking time to draw attention to that link return for you? How does it benefit you? On an objective scale I'm not aware of an ojective value other than if you choose introspection based upon my questioning what you're attracting to yourself. Thanks for your post. I've been bored by the direction those topics on randomness took as much as anyone but they created my obsession with non randomness. I'm more concerned about people finding this interesting than boring.

[onesmith 10]

You're thinking about this too much. Instead of trading what's in front of you, you're wondering whether or not it's real.

 

Just trade it, and you'll do better than most. Or overthink it and risk doing worse.

 

 

First ten bagger charting daily bars by hand one bar at a time forty years ago

[onesmith 10]

....................

 

You should have left this unedited instead of deleting it. It wasn't offensive and it wasn't off topic.

[onesmith 10]

It's not known what characteristic enabled people to determine which was real and which was random - just that they could. It was a test only to see if humans could detect the difference. I think you can still see what the test was like here:

 

ARORA

 

Random is a tricky word. Anything you do or write is somewhat random to ME. It's not random to you, because you choose it, but it is to everyone else. I say somewhat because you are still using words and sentences, there is still structure and I am familiar with it, so it's not entirely random. So when you're asking if things are random in the market, the question is 'to whom'? And how random? The markets are random to all participants, but they are not entirely random. At least that is how I see it.

 

When I go to that website and attempt to train as a guest it returns a box with nothing but a red x in the upper left corner. It's either on their end or javascript or anti-virus at my end. I have other priorities but I'm grateful for your contributions thus far because they have helped my focus. Thanks.

 

When setting upper and lower limits on a simple random number generator it taints the results. Constraints within the more sophisticated random number generators likewise contribute non-random artifacts to the random data that mimics market data. The best of the best strategies should fail when trading on random data. But imagine an algorithm generating random data. What if it's constraints are based upon the non random structure of real market data? Would the human eye be able to tell the differencer between that flavor of random data and real data? Would other strats be able to distinguish a difference in the data? Would the results of other strats improve when they were trading on this random data? Would it really be random data?

 

Is real data generated by (non random) strats?

[addchild 10]

Not answers, but various thoughts . . .

 

• If you muddle up the data in a time series (monte carlo), then the way that each member of the series relates to all other members is random, but . . . the time series will still have the same outcome, or sum to the same total. The process is random; the outcome is fixed.
  In life, order matters, (think PEMDAS). Monte carlo simulation is data in a vacuum.
   
  
• Suppose you have a known price now, and you think you can predict a future price. Assuming you can, then that still isn't the same as saying that you can predict the path involved in getting there. That would involve a hell of a lot more prediction. Which is one reason that people doing fancy things with trailing stops etc confounds me . . .
   
  
• A lot of the references to 'non-random' structures appear to be rooted purely in price, and how it relates to itself visually within a chart. Is it any wonder if such a limited view of structure should turn out to hide all manner of randomness?
  Almost all studies of randomness in markets are defined with historic price data, entirely ignoring the other side of the market, liquidity.
   
  
• What about Mandelbrot - tiny differences in even the simplest and most benign systems can have diverse outcomes over many iterations . . . and a fat-finger error from a one contract retail trader can trigger a market crash . . . Does anyone actually believe this?
  I think you are referring to the "butterfly effect"? That was Lorenz. Also, perception has leverage, If one extra contract changes the perception of someone with 10k contracts, that single lot contains a lot of leverage.
   
  
• Talk of footprints - just because an order is not randomly placed doesn't mean that the individual placing it has any kind of directional motive. If someone is operating within a completely different structure of behaviour to yourself, then are their actions really in any worthwhile sense any different to those of someone/thing behaving randomly?
   
  Markets are a function of all participants, which is independent of their motives.
   
  
• Non-random behaviour is predictable if you know the cause. Random behaviour is predictable if you know that there is no cause . . .
   
  Non-random behavior is predictable regardless of the cause. Random behavior is only predictable in the sense that it will continue to be random.
  

 

BlueHorseshoe

 

My thoughts on your thoughts.

[SIUYA 11]

What does your taking time to draw attention to that link return for you? How does it benefit you? On an objective scale I'm not aware of an ojective value other than if you choose introspection based upon my questioning what you're attracting to yourself. Thanks for your post. I've been bored by the direction those topics on randomness took as much as anyone but they created my obsession with non randomness. I'm more concerned about people finding this interesting than boring.

 

the answer to your question is largely this article reminded of the same circular boring discussions.

yeah I probably should have put it more into the other thread about randomness as everyone has an opinion on it, they all think they are right and most of the talk is rubbish. On re reading your original question - wrong thread - sorry....

 

.......

so as to your original question...

 

"Assuming everyone contributes toward creating the non random structure within the real data ......... What is the markets structure?

 

What is the non randomness that distinguishes(differentiates) real and random data?

 

It has to be human behavior and in real live data you are actually trying to pick up patterns of human behavior.

Real data does not rely on being totally random, we have biases and context, rational and irrational thought, real life news, opinions etc; etc; If you think markets bounce off support because of magic or that its random then that’s being naive. Orders in markets are based on the past for various reasons and the expectations of the future – they are not random. The patterns we choose to see might be random, but the price levels of orders are not.

I dont think it makes any difference - if the market data is random or non random we are all attempting to place it into some sort of discernible structure from which we can make sense of it, analyses it and profit from it; Which is why money management and letting a position run is important.

The only real value true (or as close to) random data has is its ability to strengthen our models of the market for robustness of that model. Most models of the market still wont make any real money, even those that make money in testing real and in random data will be likely to make less than estimated in real life, but they might give us a better expectation of what to expect should that model be applied without the added subjective input of a trader on a daily basis.

 

Maths and statistics ….. ignore it at your peril, but this is not everything. (no matter how many people try and tell you their opinion is right – relates to the article – Darwins ideas don’t set hard and fast rules but context determines a lot, ignore the predictions, look at- but -question the data and learn to anticipate what others will do)

Even the great modellers and systematic traders of the market constantly update their models or use multiple ones as the market always changes.

 

The markets structure is different therefore can be whatever it is we want to box it into – the way we see it ebb and flow and how we can then profit from that - be it that we see patterns that might be entirely random but we are still able to apply some money management, and extra context for that pattern – that’s all we need to care about IMHO and if we want to understand markets then understanding human behaviour helps take some of the randomness out of the data.

[BlueHorseshoe 10]

 

In life, order matters, (think PEMDAS). Monte carlo simulation is data in a vacuum.

 

I think outcome is far more important than process. I would rather go through hell and high water for a better outcome than have an easy ride to a mediocre result.

 

Almost all studies of randomness in markets are defined with historic price data, entirely ignoring the other side of the market, liquidity.

 

I've spent a little time looking for structure in the order book without significant success, but I have nothing authoritative to say on the topic.

 

I think you are referring to the "butterfly effect"? That was Lorenz.

 

Well, the Noah and Joseph effect also . . . Lorenz, Mandelbrot, James Yorke, Robert May . . . I read an article in Focus magazine

 

Also, perception has leverage, If one extra contract changes the perception of someone with 10k contracts, that single lot contains a lot of leverage.

 

The amount of leverage it has is random, though, isn't it? Could be 2 5k participants, or a 40k trader.

 

Markets are a function of all participants, which is independent of their motives.

 

I am not totally sure that I understand this.

 

Non-random behavior is predictable regardless of the cause.

 

So . . . I washed my hair yesterday, and I washed it the day before. My behaviour is rule governed, not random. Will I wash my hair today?

 

You need to know the cause, the rule that triggers the behaviour. Otherwise, you either have to assume that a pattern will repeat (washes hair every day) in a linear fashion, or that the bahaviour is random.

 

Random behavior is only predictable in the sense that it will continue to be random. .

 

And therefore follow certain probability distributions that are easily modelled?

 

 

That's enough over-thinking for me for one day

 

BlueHorseshoe

[Ammeo 10]

The Structure of Non-randomness is not that complicated as people think. Go Value and it'll be all fine..

[gianno 10]

What some see as random is pretty much business as usual to an experienced gambler.