The Melbourne University Seizure Prediction competition ran on Kaggle from November to December 2016, attracting nearly 500 teams. Kagglers were challenged to forecast seizures by differentiating between pre-seizure (preictal) and post-seizure (interictal) states in a dataset of intracranial EEG recordings.
In this winners' interview, the first place Team Not-So-Random-Anymore discusses how their simple yet diverse feature sets helped them choose a stable winning ensemble robust to overfitting. Plus, domain experience along with lessons learned from past competitions contributed to the winning approach by Andriy Temko, Alexandre Barachant, Feng Li, and Gilberto Titericz Jr.
The basics
What was your background prior to entering this challenge?
Andriy Temko (AT): I’m an electrical engineer and have PhD in Telecommunications and Signal Processing, previous experience with predictive modelling in acoustic and biomedical signals. This is my first competition win.
Andriy Temko on Kaggle.
Alexandre Barachant (AB): I’m an electrical Engineer and PhD in signal processing. I don’t have an academic training in Machine Learning, but I learned quite a lot thanks to various professional and personal projects. This is not my first kaggle challenge, and I consider myself as a relatively good Data scientist. Saying that, there is still a large part of the field in which i’m not very experienced (I rarely use CNNs)
Alexandre Barachant on Kaggle.
Feng Li (FL): I got my bachelor’s degree in Statistics (Xiamen University) and I’m now pursuing my master’s degree in Data Science (University of Minnesota, Twin Cities). I joined Kaggle one year ago and keep learning from various kinds of competitions.
Feng Li on Kaggle.
Gilberto Titericz Jr (GT): I’m an electrical Engineer and MSC in Telecommunications. Since 2008 I’ve been learning machine learning techniques. In 2012 I joined Kaggle and competed in more than 80 competitions and won a few.
Gilberto Titericz Jr on Kaggle.
Do you have any prior experience or domain knowledge that helped you succeed in this competition?
AT: I believe so. Knowledge of EEG processing, some understanding of seizure generation mechanisms, previous successful works on automated seizure detection for newborns.
AB : A bit. I’m a specialist of Brain-Computer interfaces (BCI) and EEG processing. I spent most of my career designing classification algorithms for EEG signals. I’m not very experienced in seizure prediction, but I participated in the two other seizure challenges on Kaggle (not very successfully). Oh yes, I almost forgot to mention that I won 3 other EEG challenge organized on Kaggle 
Fair enough, I have a lot of domain knowledge and prior experience in this area.
FL: I don’t have prior domain knowledge in EEG data. I just read relevant papers and the solutions in previous seizure competitions on Kaggle.
GT: I don’t have prior domain knowledge in EEG signals. My main experience is in using machine learning to blend models and solve the most diverse problems.
How did you get started competing on Kaggle?
AT: My first competition was a good match to my education and background. I wouldn’t have started otherwise. The current message to others would be – simple things rigorously done can bring you very far, you don’t need to have a lot of prior knowledge to participate.
AB: My first (serious) competition was the DecMeg challenge in 2014. It was pretty much the topic of my first post-doc (cross-subject classification of evoked potential), so I decided to enter and see how well the stuff I developed would do in a competitive environment. Looking back on this one, I was a very inexperienced Kaggler. I just made a few submissions and stopped working. Luckily for me, I nailed an unsupervised procedure that gave me a comfortable advance and I won the challenge with a submission made 45 days before the deadline
FL: I first heard of Kaggle in 2014 and start competing on it since 2015. When I first tried two Kaggle competitions, I found Kaggle is a great platform. People share their original ideas with each other in the forum and the winners’ solutions for each competition always impress me. In every competition I attended, I could always learn something and accumulated my experience on different kind of dataset.
GT: After Google AI challenge 2011 I was looking for other challenges when I found Kaggle. My first competition was Wind Forecasting 2012 and I luckily get a solo third place using a blend of Neural Networks models. Since then I became addicted
What made you decide to enter this competition?
AT: Good chances to win by battling others’ better predictive modelling skills with domain knowledge. I participated in the other seizure detection/prediction challenges ending up in top 10 in both of them. When I saw this one I knew I had good chances to win.
AB: As I mentioned before, I was not very successful in the 2 previous seizure competitions. The first one was after the DecMeg challenge, and I applied a similar strategy. I was on top of the LB for a couple of weeks and then I stopped working on it. I ended up 38th. I guess there is a limit to luck … lesson learned: work hard until the end and team up when you have the occasion!
The second seizure challenge followed shortly after that, so for this one I went all in. I teamed with two other very good Kagglers, and we climbed the public leaderboard up to the second place … and overfitted very badly in the process to end up in 26th place (with a drop of 0.11 AUC, hard to beat that). Thanks to this one, I really learned to fear overfitting!
So when I saw this 3rd seizure competition, I knew it was the right one for me.
FL: I need to finish a one-year long project in order to get my Masters degree. It’s a coincidence to attend this competition because I was looking for a topic which is relevant to machine learning at that time. I noticed this competition is classified as research competition which meets my project requirement. I started doing this competition at the very beginning so I have enough time to read relevant papers and previous solutions.
GT: Get some domain knowledge with the best teammates I could find.
Let’s get technical
How did you spend your time on this competition?
The first difficulty of this challenge was to build a reliable cross-validation procedure. Training and test data were recorded on two different time periods, and it was not possible to emulate this split in the CV. We had to proceed very carefully to avoid overfitting. So very roughly, here is how we spent our time in this challenge: 60% trying to find a proper CV routine. 20% trying to smile when another one was of no use. 20% modeling and feature extraction.
We ended up using multiple CV procedures, trying to gauge relative score improvement of our models at each iteration. We tried to avoid over-tuning parameters of feature extraction and modeling, while maximizing the diversity of the approach in our ensemble of models.
What preprocessing and supervised learning methods did you use?
Since we couldn’t trust our CV score, our approach was to extract as many features as possible, to build as many models as possible and finally to pick the ones that seemed the most robust and diverse. Our team had good EEG domain knowledge, we also reviewed top 10 solutions of the 2 previous seizure challenges as well as the literature. That gave us a solid list of feature to chose from (FFT, correlation, coherence, AR error coefficient, etc …)
We then trained a bunch of different classifiers (XGB, SVM, KNN, LR). We limited ourselves to simple solutions with minimal parameter tuning. In this regard, Gilberto’s experience in guessing (almost-optimal) XGB parameters from the first iteration was a big deal.
What was your most important insight into the data?
The dataset was small, noisy, and CV was unpredictable and unreliable - It was a very challenging problem. So here is our more important insight: Diversity in the ensemble is the key to robustness. We use many simple and relatively low performing models rather than trying to hyper-optimize our best performing models (and overfit in the process). Simple feature sets works very well in this dataset. On the contrary, the performance drops down when we include more complex features sets.
Figure 1: Correlation map between each of the 11 individual models and the winning solution. The overall low correlation show a strong diversity in the models predictions.
The second most important insight came from experience when we had to pick our 2 final submissions. We believed the public leaderboard to be very overfitted (including our best public score) and that the private score will settle down around 0.8 AUC. We decided to minimize our risk by choosing as one of our final submissions a very conservative ensemble, tailored to be stable. It was a difficult choice, the public score of this ensemble was around 0.81 while our best score was 0.85. Luckily enough, our prediction was right and everyone went below 0.8 AUC on the private LB except for our stable ensemble that barely moved.
Which tools did you use?
Python, R, Matlab. All with various modules/toolboxes.
What was the run time for both training and prediction of your winning solution?
Ranging from a few hours to a few days (mainly due to the usage of Matlab and old toolboxes).
Words of wisdom
When CV is unreliable, don’t panic, simple things and basic ensembling (and teaming) provide a very stable solution.
Teamwork
How did your team form?
Contacting people by identifying those whom you can learn from and who can help the team to win.
How did your team work together?
Lots of chit-chatting, brainstorming, “a problem shared is a problem halved”.
How did competing on a team help you succeed?
Teaming brings together much more human resources and brainpower to solve a problem at hand. Eventually, it allows for a combination of many different solutions/directions that one alone would not have thought of. Last but not least, it allows to keep a strong motivation even when you are stuck and nothing seems to work.
