The Integer Sequence Learning playground competition, which ran on Kaggle from June to October 2016, was a unique challenge to its 300+ participants. Given sequences of integers sourced from the Online Encyclopedia of Integer Sequences, the goal was to predict the final number for each among the hundreds of thousands of sequences.
In this interview, Gareth Jones and Laurent Borderie (AKA WhizWilde) of Team 1.618 describe their approach (or rather, approaches) to solving many "small" data problems, how they would have used unsupervised methods to cluster sequence types were they to start over, and their advice to newcomers to machine learning.
The basics
What was your background prior to entering this challenge?
Gareth: I’m a post-doc at the UCL Ear Institute, UK working on multisensory decision making and perception. I started as an experimental neuroscientist but have progressively become more computational.
Laurent: I work as a biological data analyst, switching from computational biology to bioinformatics and machine learning as needed. I am a pharmacist specialized in Research with Masters degrees in Neurobiology/pharmacology and in Project Management and Therapeutic Innovation in Biotechnology, but I became a computer scientist from studentships, past jobs, many MOOCs and out of personal interest for coding, machine learning, maths and AI.
Do you have any prior experience or domain knowledge that helped you succeed in this competition?
Gareth: Not specifically with integer sequences, but I have experience working with handling and analysing large datasets and machine learning in general.
Laurent: I love maths and am always willing to discover more about advanced topics. Plus, analyzing data is like an occupational hazard for me and I have been involved in a few Kaggle competitions.
How did you get started competing on Kaggle?
Laurent: Through No Free Hunch. I wanted to test myself and I got hooked.
Gareth: I have completed a number of Coursera and edX courses over the past few years on machine learning, neuroscience, and finance. Kaggle was the next logical step in trying out the things I’ve learned.
What made you decide to enter this competition?
Laurent: As I said, I love maths. Plus, it has a scientific interest, and also, it was a change from the usual “data table format” which I had mainly approached.
Gareth: It’s an unusual and tricky problem from a machine learning perspective, one that really highlights the importance of always remaining skeptical of models that superficially fit well.
Let’s get technical
What preprocessing and supervised learning methods did you use?
Gareth: Very little and only really linear regression. No major preprocessing was necessary for the approaches I tried, and, I think, this competition could require no supervised learning at all. Ultimately, most of my solutions were linear fits to a rolling window of previous points (Figure 1) (~37,000/115,000), however only around 15% of these were actually correct. The valuable solutions came from specific solvers that employed methods such as common differences (~15,000 sequences), pattern search (~2000), recurrence relation (~10,000), etc. The solvers I implemented are described in more detail here.
For sequences that were predicted using regression, the training data was generated from the sequence itself, rather than from the separate training set of sequences provided (Figure 1). A rolling window of a set number of previous terms of the sequence was applied, with the next term being the target. The last known term in the sequence was held out, and as many windows generated as possible with the remaining terms. The model was trained on this data and performance was assessed as % difference between the next prediction and the held-out term.
I also tried fitting SVMs using the same method to try and avoid imposing a parametric function on each sequence, but there wasn’t enough data available in the known terms of each sequence to train these successfully.
Figure 1 - Generation of training and validation data for sequences fit with linear polynomials. Training data used a set number of previous points with each term in the sequence being the predictive target. The final known term was used to validate accuracy of the model.
Laurent: My preprocessing was only meant for two things; either to facilitate sequence handling or allow for specific operations like scoring prediction through similarities or differences between tested and model sequences. Tagging also allowed using specified fallback methods in case no prediction could be made (either no prediction available, or depending on the setting, score under a threshold). I used a little linear regression, for some predictions. My approaches are detailed here.
I also threw the bases of an ML approach to learn from both general and positional tagging, but had no time to pursue it. I wanted to learn from each sequence as its own train set, or from classified/tagged parts of the set or from whole train set with both supervised and unsupervised methods. Tools I was interested in using were LSTMs, genetic algorithms in addition to more regular forests, Neural Nets, or SVMs.
What was your most important insight into the data?
Laurent: I had no special “revelation” about the data, though I had various ideas about how to approach the problem, which were mostly proven relevant. I used various signatures methods; model sequences set improvement by sequence reversing, shifting, merging; by addition of commonly known sequences; sequence tagging (I came with my own and added Gareth’s to it); specific scoring approaches. I knew that technical aspects (handling large numbers, efficient and flexible structures) would be crucial, and that sometimes mere statistics could be better that stretched predictions, because of inference problem (a “right” prediction for given sequence could be wrong for this sequence number, which address a specific problem in OEIS your methods are not understanding correctly).
Gareth: Two things:
- That sequences can be grouped by their fundamental properties to inform solver selection, which I unfortunately never got around to integrating with my methods, but I think could lead to a more intelligent approach to the problem.
- This problem isn’t one big data problem, rather it’s many small data problems. Each sequence is independent of all the other sequences, meaning that training of a model needs to use the inter-sequence data only, which is very limited. Another difficulty comes with scaling these small solutions to apply to hundreds of thousands of sequences without simply generating a harmful number of false-positive solutions. Fitting complex functions with very limited validation risked creating false positives at a harmful rate. False positives prevent fallback to sequence mode (the last-resort solution), and despite the mode being generally very poorly predictive if the next sequence term, it provided more value than trying too many fits.
Were you surprised by any of your findings?
Gareth: I’m a bit surprised how low the scores are overall, I don’t think we’ve totally cracked this problem yet.
It was possible to cheat in this competition by simply looking up answers on the OEIS, so I’m hoping others with (realistically) high scoring submissions will share their solutions – I’d like to know what else worked!
Laurent: After the competition, I submitted the results of a script that I could not merge in time with our results because of technical problems. It was quite poor alone (0.0722), but it actually boosted our results by 0.00586 from 0.21939 to 0.22515 by merely permuting its values with equal-to-zero prediction into our final submission, meaning it was an entire new set of mostly good predictions (despite losing some fraction of points by changing some solutions to wrong predictions) I am still running scripts with new twists to check and try to improve this.
Which tools did you use?
Gareth: R and RStudio.
Laurent: Python/Anaconda suite, plus a few unusual packages like Sympy, which allowed me to work with primes and some types of sequences.
How did you spend your time on this competition?
Gareth: Most of my time was spent trying to find and implement general methods to solve sequences. The rest was spent on trying to figure out which solvers (including ML techniques) were most reliable and in what order to apply these in to limit false-positive solutions.
Laurent: I employed time reading about the problem/advanced math topics, coding and optimizing processing and algorithms to be able to handle the enormous load that could represent using some methods, finding methods and testing which would have the best return and optimizing scoring for improving predictions in the end.
What was the run time for both training and prediction of your winning solution?
Laurent: From maybe 2 or 3 hours with reasonable settings to forever and a day when trying “brute force” approaches.
Gareth: Without any fitting, only around an hour. With linear regression, maybe 12 hours. With attempts to fit non-parametric or non-linear models (which never actually helped), considerably longer!
Words of wisdom
What have you taken away from this competition?
Gareth: That if I were to start again, I’d totally restructure my assumptions to frame the problem differently.
Everything I did was based on an early assumption that the reliability of a solver would be the same for all types of sequences, meaning solver priority would be constant across sequence types. However, this is a gross oversimplification; consider, for example, mode-fallback (simply taking the mode of the known terms to predict the next). On average, it’s a very poor solver, but for a binary sequence where 99% of the terms are 0, it’s almost always correct. Conversely, there’s far less chance the mode will be correct for monotonic sequences.
A better approach might be to tag the basic properties of sequences, use some unsupervised learning to cluster types of sequences (Figure 2) and then to test solver reliability on each group. It would then be possible to dynamically apply solver priority based on these basic properties of a sequence. Perhaps it would even be possible to use the groups as training data, making this problem more accessible to more supervised learning techniques.
Figure 2 - Clustering of the tags given to 30 randomly sampled sequences, with examples from high level groups. Sequences from different groups are likely to have different classes of generating functions.
Laurent: As Gareth, I would maybe do the things differently.
- I wanted to test a lot of things, but could only work on this during a period of 5 weeks. I focused on what appeared the most efficient, the use of solvers based on similarities and methods depending on algebra, and mostly integrated scripts that could run without my attention. But instead, I sometimes would have tested some individual approaches on the whole set in dedicated scripts, and I would have used their results to inform more accurately my next steps. Here it was more difficult to do, as I knew I could only have a shot at a main method with few defaults, so instead I wanted to give it the most power by allowing it to go to the limits.
- I came with nice scoring but I also would have liked to better mix such methods with statistical and machine learning approaches, with improved tagging with ML methods in addition to tagging through algebra tests, and with a ML system to optimize scoring settings.
- I would test more variants of my twists, too, like more complex sequence merging.
- I would also anticipate scaling problems by coming with better methods to reduce processing to chunks, as method which proved fast and efficient with small and medium number of sequences unexpectedly proved absolutely impractical in the end.
Do you have any advice for those just getting started in data science?
Gareth: Always try and apply the things you learn; MOOCs for theory, Kaggle for practice. Also, try to get experience collecting data as well as analysing it. Experimental science is difficult and messy in ways that aren’t obvious simply by looking at a dataset. Understanding how and why will give will you a healthy skepticism for the quality of all data.
Laurent:
- Don’t be shy, fearful, or discouraged by your lack of formal (or informal) knowledge. ML can be impressive and challenging as you discover it, but to become familiar with its fundamental concepts, you just need to toy with it, to play with algorithms and see the results, even before knowing their deep concepts. And it’s even easier today with suite and programming languages made just for this. You can learn what you need to challenge real theoretical walls in time.
- When you learn, don’t lose yourself in books or MOOCs, choose one adapted to your path and go from start to the end, then toy again, practice, and perform in competitions or on real life problems.
- Enjoy yourself. Think outside the box. Redefine the box. Read and talk to people on the forums
- Take your time. I know there are Kagglers who took only a few months to become masters from scratch, but no need to rush. You just want to be skilled in the end, and there is no shortcut to your own path.
Teamwork
How did your team form?
Gareth: Laurent and I discussed a lot of interesting ideas on the forums throughout the competition and decided to team up to combine our approaches.
Laurent: I appreciated Gareth’s clever insights and I felt it was very easy to understand him and reciprocally, so teaming only felt natural as it would allow sharing things more freely and efficiently. We just had to find a clever name and we were done.
How did your team work together?
Gareth: We teamed up fairly late in the competition and were working on different approaches in different scripting languages. Rather than using time converting, we generated submissions and combined them based on how reliable we estimated the methods available to solve each sequence were.
Laurent: We shared views on options and results selections, also, he ran some of my scripts that my computer couldn’t handle as they felt promising. Had we had worked together from the beginning, I guess it would have been a little different.
How did competing on a team help you succeed?
Gareth: Laurent implemented a number of difficult and valuable methods that I would never have had time to alone. Our discussions throughout the competition, even before teaming up, stimulated a lot of ideas.
Laurent: Well, Gareth already had good results by himself. I merely added a few things on top of that in the end. Some post-competition submission showed that I could have contributed even more to the final results. Sharing in a team is more fun, it allowed us to find better ideas. We could share hardware resources, which proven handy, too.
Just for fun
What is your dream job?
Laurent: The same as now, with more people from various fields around me, with people to admire and others to coach, and more projects in ML.
Gareth: Something involving neuroscience and machine learning - either using machine learning to guide health decisions (like in the EEG seizure competition, which is currently taking up my spare time), or, conversely, using neuroscience to inform the development of machine learning algorithms and decision making in AI.
Bios
Gareth Jones has a PhD in Neuroscience from The University of Sussex, UK and is currently a post-doc at the UCL Ear Institute, UK. His research uses electrophysiology, psychophysics, and computational modelling to investigate the neural mechanisms of sensory accumulation, multisensory information combination, and decision making.
Laurent Borderie works as a biological data analyst, he’s a PharmD, with Masters in Neurobiology and Biotechnology Therapeutic Innovation. He learned ML besides his academic curriculum, and his research interests includes Artificial intelligence, Sentiment and behavioral analysis, as well as brain computer interfaces, and biological analyses interpretation heuristics.