Data Mining with R and Rapidminer

[darthtrader 10]

From the thread that brownsfan and I posted in about Steenbarger and his volume/days range stuff until now I've stumbled across some AMAZING stuff in this area. To program alot of this stuff in stock software is to reinvent the wheel.

 

This interesting thread came up on Elite that mentions Rapidminer.

http://elitetrader.com/vb/showthread.php?s=&threadid=117361

 

In trying to find information about it I finally installed and actually loaded some YM data into the heavy weight open source statistical computing environment, R. Its amazing how simple it makes tasks that are far beyond most stock software analysis wise.

 

Here is an entire course on data mining with R

http://www.stats202.com/

The videos for the class are all up on google video if you search for data mining + long format.

 

This is an entire free book on data mining with R, 2 different hands on projects and one is a forecast for IBM stock prices using nets.

http://www.liaad.up.pt/~ltorgo/DataMiningWithR/

 

this has a 5 part video tutorial on using Rapidminer:

http://www.neuralmarkettrends.com/tutorials/

 

Jerry in that elite thread had a project idea for rapidminer and I had just sent him a message inviting him to bring it to TL. The premium section here would be quite an ideal place for such results.

 

is anyone else interested in this stuff? While I'm not interested in "prediction" it just seems like a huge waste of time to not use these tools to find relationships that would take years of experience to uncover(if ever).

[darthtrader 10]

Here is possibly an even better piece of software.

http://rattle.togaware.com/

 

Its basically a java GUI built on top of R. Sounds like it makes it easy to explore various mining algorithms but then also spits out R code to learn from once you want to progress beyond whats already been built.

[mister ed 10]

Hi Darth - this does sound like good stuff - will take a good look over the next few days - thanks for the links

[darthtrader 10]

Well I have already given up on Rapidminer. While extremely interesting there simply is no way documentation wise that someone with no background in stats/data mining is going to be able to get up to speed with it.

R is a different story having entire series of textbooks written on its use from introductory stats to extremely complex stuff.

 

I've just ordered this textbook which sounded like a good way to get up to speed on the various mining algorithms

Data Mining: Concepts and Techniques, 2nd ed

http://www.amazon.com/Data-Mining-Second-Techniques-Management/dp/1558609016/ref=pd_sim_b_3

 

I'm still trying to figure out what R book to go with but this one sounds pretty nice, focusing on working code and available data sets to learn R with.

http://www.amazon.com/Statistics-Introduction-Michael-J-Crawley/dp/0470022981/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1209041520&sr=1-1

[darthtrader 10]

Just some general R resources before I lose them.

 

RExcel is probly the way to go at this level. You can basically either use R within excel or read and write excel files directly in R. Should be able to get R doing real time market stuff with a data providers DDE link with this.

http://sunsite.univie.ac.at/rcom/server/doc/RExcel.html

 

Good R learning resource

http://www.mayin.org/ajayshah/KB/R/index.html

http://www.agr.kuleuven.ac.be/vakken/statisticsbyR/

 

good financial stuff in R

http://www.burns-stat.com/

 

charts for R

http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=65

 

financial engineering stuff

http://www.rmetrics.org/

[GammaJammer 10]

ok - new to data mining (although not new to the concept, or to the markets). Is there anything online you can reccomend for me if I'm lookign to really start from scratch. Can program in VBA so I'm sure I could get my head round the code if I try.

 

Any / all answers appreciated. Have a tactical relative value trading product here which I am spearheading (in between doing a grillion other things as per usual), so might be getting involved in this a bit. Been looking at all sorts of options. FX engines etc etc. But while there are plenty of pruducts out there to backtest stuff for you if you are specific enough, ultimately what I really think I have to do is do the data mining myself.

 

And for both cost and time reasons I'm thinking it might be easiest to actually do it myself (or at least, supervise while ony of our jnr traders does the grunt work) than pay for an engineer to come in and do it (especially as I'm not 100% sure what I'm looking for, just have a few ideas I want to look at right now and thats never an efficient use of outside IT type people in my experience).

 

Sorry this post is a bit rambling - busy afternoon on the desk so I keep coming back and adding a bit

 

GJ

[lote_tree 10]

I'm actually looking to do some data mining myself soon. I have JMP 7 by SAS. It's an extrodaniry piece of software to mine large amounts of data:

 

http://www.jmp.com/software/jmp7/

 

I use Investor RT and the only problem I'm having at the moment is figuring out how to extract 5 years worth of Market Profile pivot data for the E-Mini S&P. If anyone can help do this I can hopefully post some results for people in the premium forum.

 

Cheers.