The Bosch Production Line Performance competition ran on Kaggle from August to November 2016. Well over one thousand teams with 1602 players competed to reduce manufacturing failures using intricate data collected at every step along Bosch's assembly lines. Team Data Property Avengers, made up of Kaggle heavyweights Darragh Hanley (Darragh), Marios Michailidis (KazAnova), Mathias Müller (Faron), and Stanislav Semenov, came in third place by relying on their experience working with grouped time-series data in previous competitions plus a whole lot of feature engineering.
The banner image is from gingerman's kernel, Shopfloor Visualization 2.0.
The Basics
What was your background prior to entering this challenge?
Darragh Hanley: I am a part time OMSCS student at Georgia Tech (focus area Machine Learning) and a data scientist at Optum, using AI to improve people’s health and healthcare.
Darragh on Kaggle.
Marios Michailidis: I am a Part-Time PhD student at UCL, data science manager at Dunnhumby and fervent Kaggler.
Marios Michailidis on Kaggle.
Mathias Müller: I have a Master's in computer science (focus areas cognitive robotics and AI) and work as a machine learning engineer at FSD.
Mathias Müller (AKA Faron) on Kaggle.
Stanislav Semenov: I hold a Master's degree in Computer Science. I've worked as a data science consultant, teacher of machine learning classes, and quantitative researcher.
Stanislav Semenov on Kaggle.
How did you get started with Kaggle?
Darragh Hanley: I saw Kaggle as a good way to practice real world ML problems.
Marios Michailidis: I wanted a new challenge and learn from the best.
Mathias Müller: Kaggle was the best hit for “ML online competitions”.
Stanislav Semenov: I wanted to apply my knowledge in practical problems.
Do you have any prior experience or domain knowledge that helped you succeed in this competition?
We don’t have any such domain knowledge, but already have big experience with grouped time-series data (like in RedHat or Telstra Competition) to generate right features.
Let's Get Technical
Features
There was a lot of feature engineering.
Early in the competition it became clear there was a leak in the data set which only a few teams had found. Soon after, Mathias found the leak and released the magic features in a public kernel. The leak involved sequential components with the same numerical readings having a high rate of failure. The public release of these features opened the competition and remained our strongest feature throughout.
We also found extra information by using different criteria for what are sequential components. With so many different production lines and stations, two components could be sequential at one station, but then pass through different stations for the next phase of production. Given this we identified sequential components, date-wise, within each individual station with the same numerical readings. We found some stations, such as L3_S29 and L3_S30, worked particularly well as most components passed through these stations. This can be seen particularly well in John M's visualization of the manufacturing process. After identifying this, we could build on it by counting how many stations a pair of components had the same values in, or by counting the number of times those numerical reading occurred over the whole data set.
John M's kernel visualizing the manufacturing process.
We also saw varying trends in failure rates over time, both in the short and long term horizon. We trained models using rolling mean of the component failures sorted based on start and end dates of all stations. We calculated using different rolling windows – 5, 10, 20, 100, 1000, 5000 components – to catch both the long term and short term trends in failure rates. It was important to calculate such features out of fold to prevent overfitting. Below can be seen OOF rolling mean compared to usual rolling mean with a window size of 5 components.
OOF rolling mean compared to usual rolling mean with a window size of 5 components
Similarly we could capture the lag and lead of the target out of fold, something which tree based models would not capture well out of the box.
Besides this, we had a lot of usual features:
- Encoded a few categorical columns with out of fold Bayesian Mean
- Counts of non-duplicated categorical columns, counts of non-duplicated date and numeric columns.
- Encoding the paths of components through the stations; whether components passed through the same sequence of stations.
- Row wise NA counts for numeric, as well as max/min per stations.
Overall, we have more than 2000 features at 1st level models.
Validation
We used a 5-fold cross-validation. Unfortunately, our validation improvements have not always coincided with improvements on the leaderboard because of the discrete metric, Matthews correlation coefficient.
Models
There were about 160 models on the 1st level. Most of them are XGBoosts, but in this competition LightGBM has proved to be very good. We also have Extra Trees Classifiers, Neural Nets, Random Forests and Linear Models.
Meta-modeling
Meta modelling was really simple in this competition (compared to other competitions). We just used one bagged XGBoost on the 2nd level. Model selection for meta was done based on the same 5-fold cross-validation that we used at the base level. After model selection we had a total of 45 models. Any other stuff did not work.
Schematic of meta-modeling used.
Final Ensembling
In this competition we should predict strictly 0 or 1 – so in choosing a probability threshold, rows close to the threshold showed a lot of randomness – sometimes being predicted as 0, sometimes as 1. To mitigate against this, we used a majority vote of a number of different discrete predictions as our final selection. The result gave a nice boost - around 0.003 on public and private LB.
Below you can see a histogram (cut off on the y-axis) of the row wise sum of the 25 subs with some information on how many rows were always predicted as 1 or the other class or sometimes both classes across different submissions:
Histogram (cut off on the y-axis) of the row wise sum of the 25 subs.
Bios
Darragh Hanley (Darragh) is a Data Scientist at Optum, using AI to improve people’s health and healthcare. He has a special interest in predictive analytics; inferring and predicting human behavior. His Bachelor’s is in Engineering and Mathematics from Trinity College, Dublin, and he is currently studying for a Masters in Computer Science at Georgia Tech (OMSCS).
Marios Michailidis (KazAnova) is Manager of Data Science at Dunnhumby and part-time PhD in machine learning at University College London (UCL) with a focus on improving recommender systems. He has worked in both marketing and credit sectors in the UK Market and has led many analytics projects with various themes including: Acquisition, Retention, Uplift, fraud detection, portfolio optimization and more. In his spare time he has created KazAnova, a GUI for credit scoring 100% made in Java. He is former #1 Kaggler.
Mathias Müller (Faron) is a machine learning engineer for FSD Fahrzeugsystemdaten. He has a Master's in Computer Science from the Humboldt University of Berlin. His thesis was about 'Bio-Inspired Visual Navigation of Flying Robots'.
Stanislav Semenov (Stanislav Semenov) is a Data Scientist and Quantitative Researcher. He has extensive experience in solving practical problems of data analysis and machine learning, predictive modelling. Co-founder of Moscow ML Training Club, World Champion of Data Science Game (2016). At the current time, he holds 1st rank on Kaggle.