Updated Prediction app

Noise

Over the last few months I have been developing a new application.  It's purpose is to visualize Premier League trends and predictions by leveraging the freakishly awesome D3.js libraries found at http://d3js.org/.  I call the app noise, as it is intended to both counteract the noise associated with sports punditry, while recognizing the inherent complexity of the tracking and measuring team performance and fan engagement. You can view the site here.

I have built two features, with a third feature in the works.  The first is a simple social media tracker (in this case Twitter- I plan to add additional sites such as reddit in the near future). This provides a snapshot of the current volume of information being exchanged concerning a team. 

Additionally I have trained a statistical model to predict the outcome of each match.  The model is built by comparing the home teams home performance: average wins, goals, corners, and shots on target against the away team's away performance across the same metrics.  Additionally I calculate both teams statistics as an expanding average over the season, and a rolling average for the last three games. 

If any one is interested in the model used, or wants to add something to it the code, the project is located on github.

New EPL predictions

I spent the last week and a half reworking the model and building out the skeleton of a web app to better convey the results. I will expand on the web app in the near future to include a more holistic, unsupervised look at the match.

http://noise-roc.appspot.com/prototype

For those interested - the tools in use here include Python for the data munging, Orange (a python ml library) for the modeling, and d3.js for the visual.

Premier League Predictions

This is my first attempt at using machine learning to predict EPL matches. There are significant  improvements to be made - which I will gradually incorporate in future updates.

A brief walkthrough of the visual - the model results describe the likelihood of a favorable outcome - zero representing a low probability of success. The 365 Odds and the data for the model itself can be sourced from this website.

Arsenal and Tottenham - Full

The tweets are aggregated by positive and negative sentiment

Twitter During the Arsenal Game on Saturday

A visualization of twitter and the English Premier League:

Extracting Sentiment

This is just a quick follow up to the twitter sentiment visualization. The following is a description of some of the technical challenges I faced.  This is by no means a complete analysis - some might even consider it naive - but the purpose here is not to build the worlds best twitter analyzer  but rather to build a framework with which one can extract tweets, process them and begin to derive meaning. 

The tools in use here include:

Python, Twitter API's, NLTK, a word sentiment corpus (I am using the one available via the Coursera Data Science course), and for the visualization I am using Nodebox

I began by extracting tweets - here I pretty much just followed the instructions on the Courersera Data Science course - for detailed steps on setting up the Oath2 protocol and the necessary dependencies on a Mac - check out this earlier post.  

Once I had tweets, I had to normalize them. Tweets are messy-they feature an extravagant use of vowels, non standard English, and special characters:

    #Convert to lower case

    tweet = tweet.lower()

    #Convert www.* or https?://* to URL

    tweet = re.sub('((www\.[\s]+)|(https?://[^\s]+))','URL',tweet)

    #Convert @username to AT_USER

    tweet = re.sub('@[^\s]+','AT_USER',tweet)

    #Remove additional white spaces

    tweet = re.sub('[\s]+', ' ', tweet)

    #Replace #word with word

    tweet = re.sub(r'#([^\s]+)', r'\1', tweet)

    #Trim

    tweet = tweet.strip('\'"')

   #Check if the word starts with an alphabet
    val = re.search(r"^[a-zA-Z][a-zA-Z0-9]*$", word)

   #Look for a patter of 2 or more letters and replace with the         character itself
    pattern = re.compile(r"(.)\1{1,}", re.DOTALL)

---

Because of I have no idea how to format code for a blog, I will refrain from pasting in code here, but instead just describe - in detail- the process.

Once I have a "clean" tweet, I use the following steps to process it:

1. First I remove all "stop words" so - words in ('is', 'are', 'the', ... ) basically any word that has no inherent emotional value is removed.  While omitting stop words, I match the tweet against the word sentiment corpus mentioned earlier, and, based on the total sentiment value of the tweet I assign it a 'positive', 'negative', or 'neutral' value.  This was my hacked up way of coming up with a training set, or examples with which to build a model and could use a lot of improvement.  More on that in the future.
2. I take all tweets that have either a positive or negative sentiment and a geotag and append them to a list
3. Now it is time to use the NLTK tools to extract a feature list.  For more on that - see their documentation here.
4. With features in hand, you can go ahead and train a classifier - a good example of this can be found here.

Once I have a satisfactory classifier, I store the model using pickle(), and start classifying new tweets. In the coming weeks I will upload the full code (a hack job if there ever was one) to github.

An average day on twitter

Twitter on July 24 2013

Twitter on the Royal Baby

I just completed a trial twitter sentiment analysis.  The larger circle represents the positive tweets associated with #Royalbaby, the smaller circle represents negative sentiment. I used python to extract an hours worth of tweets and Nodebox to construct he visual.

Thank you JJ

Finally a solid scifi concept that balances fun, action, and philosophy into a visually amazing package. The bonus? It is accessible to people who adhere to accepted hygiene practice's, and have plans for the weekend that don't involve the words "game" and "workshop". A solid movie- looking forward to seeing it once more.

Coursera - Data Science 101

I am participating in the data science course, freely available at coursera.com.  I wanted to set up the environment on my mac, and bypass the virtual machine environment (I hate working on a virtual machine).  Here are some of the extra steps needed to get the course working on a Mac, I am using snow leopard.

1. Download and install python 2.7 form the python website
2. You will need to install oath2-1.5.211 in order to access the twitter stream.  Download here
3. Install the new library by navigating to the directory of the file "setup.py" inside the oarth2 folder in the command line and typing: sudo python setup.py install (enter your password when prompted)
1. I received an error at this point complaining about not being able to locate the setuptools package.  If you also see this error, use the following steps to rectify:
1. Search for and download setuptools-0.6c11-py2.7.egg
2. In the command line run sudo sh setuptools-0.6c11-py2.7.egg (password again)
4.  Once the setuptools has been installed, try the installation of the oath2-1.5.211
5. Follow the rest of the directions as outlined on the course website.

Hope this helps anyone who had trouble! 

/a week for work and ransac

Too Much Work

My first week back at Big Blue in 2013

Last week was my first week back at big blue in 2013, and as one can imagine I was fairly busy playing catch up   Mostly model production readiness tests, sprinkled with some meetings and emails.  Production readiness in this case entailed pushing as much code to run  "in database" as possible, and there was a lot of code. Go postgreSQL.....

/ransac

As a result of stupid hours put in at work, I  contributed only minimal time to iMobi.  The video tutorial (partially complete) being one, and an algorithm called ransac: http://en.wikipedia.org/wiki/RANSAC that uses the Kinect's point cloud to identify planes, and hopefully where the floor is, being the other.

More on those two next week.

Modeling American Football

/the Problem

Analysts in my company received a challenge to build a model that can predict wins and losses in the NFL.  

Gain Understanding

A crucial step

Knowing absolutely nothing about the sport, I decided to try my hand at the problem; how hard could it be? My first task was to collect enough data for training, test and validation sets. I started by extracting outcomes for the 2008 - 2011 seasons from the website: Pro-Football-Reference.com.  Using the same website I also gathered basic statistics for each team.  The variables created within the SRS (Simple Rating System) which calculate team offensive and defensive strength relative to average NFL team performance became the first data points. Before any modeling could take place, I needed to understand the game more. I spent a couple hours reading blogs on what statistics are the best representation of a teams likely performance, and a couple more hours just reading about various aspects of the game.


In an ideal world my model would include individual player level data, but the scope of this type of collection exercise quickly exceeded my available bandwidth.  Instead I decided only consider aggregate team statistics. 

The term "SRS" seemed to pop up a lot so I wanted to start there.  I organized each training example into a Y vector that contained the outcome of each game.  Next I transposed the data I had gathered into a matrix where each column represented a feature: SRS, SoS, OSRS, DSRS etc.  Each row in the matrix contained the aforementioned statistics for each team in a game.  So one row might look like T1_SRS, T2_SRS, T1_Home, T2_Home etc.

Exploration

Follow the white rabbit

I tried several variations of two machine learning algorithms: first I constructed a decision tree that uses the  Information Entropy to prune. Next I tried a neural network.  I have coded a custom neural network that uses backpropogation to learn the weights, but for exploration I just threw it into SPSS.  I like SPSS because it is user friendly and I can get results quickly, but I also found myself severely limited by the software.  For example, I was unable to add more then two layers, and I did not see a way to play around with the bias term.  That being said the neural network outperformed the decision tree algorithm so I decided to to work to refine the model using neural networks.

While I was able to obtain a 93% "accuracy"  on the Training and Test sets, I only saw a slight improvement over a naive model with the holdout sample.  This was proving to be more difficult then I first imagined.  I quick look at the learning curves revealed the problem.  I was severely over fitting the training data, this is a problem often caused by "variance"  or "noise" in the target and sometimes the quickest way to improve the model is to gather more data...so back to google.

First I extracted three additional years of data from the initial website, next I pulled in the vegas odds on each of the games hopping to make use of the professional betting establishment's sentiment.  Finally I dug deeper into the football blogosphere and came across the website: http://www.advancednflstats.com/.  Which includes both team efficiency ratings and predictions in an easy to extract format. The inclusion of this new data gave my models a significant boost, and I am now correctly identifying 100% of the cases in the training and test sets and percentage on the validation set good enough to go to Vegas with.  

Next steps

Refinement

There is likely a lot of co-linearity in the data, I want to reconstruct the model using my neural network in Octave so I can have more flexibility with the architecture and try non-supervised ML techniques to address the collinearity.  Hopefully these steps will improve the model performance. 

a new sensor

/theKinect

First Impressions

I just received the Microsoft Kinect for Developers from Amazon.  I intend to use it as the primary sensor for my robot; It has a lot of interesting features including an IR sensor, depth sensors, audio, camera, and a fantastic set of API's.  For the cost -200.00 U.S. dollars- it seems like the easiest place to start. After the unboxing I downloaded the Kinect SDK and Developers Toolkit 
-http://www.microsoft.com/en-us/kinectforwindows/develop/developer-downloads.aspx-
and spent a couple days playing with the code.  The two main development languages are C# and C++ and are numerous examples using both - I chose C# to get started. 


A side note, I have never coded in C# before, but I was able to get up to speed with the help of some tutorials posted by Microsoft 
-http://channel9.msdn.com/Series/C-Sharp-Fundamentals-Development-for-Absolute-Beginners. 
I spent a couple hours and was able to figure the rest out from there.

My first step was to just turn the sensors on and load the video into a window; next I played around with the face tracking API's. In all it was fairly easy to get things up and running, and to start doing some really cool thins with the data streams the Kinect has on offer.  Latter on I will post some videos with actual code.

iMobi

Winter [vacation] is coming 

iMobi- This idea has been forming over the past two years.  I took the +Coursera +Machine Learning  course  by Andrew Ng and latter two +Udacity courses (AI for robotics, and CS101).

---

Concept

A to B to C 

A robot that can sense it's surroundings, build models based on those sensor inputs and make decisions based on those models.   I want  to drop it in a room, or outside in a desert or another planet.....just kidding, but that would be cool...  and it should start learning about it's environment.  I have no idea if I can ever finish this alone, but I hope my rather simplistic outline will provide a structure for the systems I will need to develop.  

Here is what I have in mind:  Picture a human brain and it's perceptions of the world.  It is able to perform  relatively simple tasks such as recognizing a handwritten "t"  to fairly complex tasks like understanding irony, or making an apple pie.   One might be tempted to think the brain's circuitry operates on a scale where less complex tasks are a assigned to less complex circuitry while the more complex tasks are left to complicated computational unites.  As it turns out this is not the case; each computational unit in the brain is no more or less complicated than the other- the brain makes sense of the world by layering simple computations. The computational unit that recognizes irony relies on input from dozens of other units that have already done their job. something had to process the visual images, something had to recognize a spoken word, something had to recognize grammar etc. Each unit performs it's task and eventually we understand irony.

I will borrow an idea from +Ray Kurzweil's book, How to Create a Mind, and call these computational units recognizes. In a similar way, perhaps the only path from input to model to decision is to take a page out of natures book and construct separate model's or "recognizes" for different tasks.