The Allstate Claims Severity recruiting competition ran on Kaggle from October to December 2016. As Kaggle's most popular recruiting competitions to-date, it attracted over 3,000 entrants who competed to predict the loss value associated with Allstate insurance claims.
In this interview, Alexey Noskov walks us through how he came in second place by creating features based on distance from cluster centroids and applying newfound intuitions for (hyper)-parameter tuning. Along the way, he provides details on his favorite tips and tricks including lots of feature engineering and implementing a custom objective function for XGBoost.
Background
I have MSc in computer science and work as a software engineer at Evil Martians.
Alexey on Kaggle.
I became interested in data science about 4 years ago - first I watched Andrew Ng’s famous course, then some others, but I lacked experience with real problems and struggled to get some. But things changed when around beginning of 2015 I got to know Kaggle, which seem to be the missing piece, as it allowed me to get experience in complex problems and learn from the others, improving my data science and machine learning skills.
So, for two years already I’ve participated in Kaggle competitions as much as I can, and it’s one of the most fun and productive pursuits I’ve had.
I noticed this competition during the end of Bosch Production Line Performance, and I became interested in it because of the moderate data size and mangled data, so I can focus on general methods of building and improving models. So I entered it as soon as I got some time.
Data preprocessing and feature engineering
First, I needed to fix skew in target variable. Initially I applied log-transform, and it worked good enough, but some time after I switched to other transformations like log(loss + 200) or loss ^ 0.25, which worked somewhat better.
Target variable and its transformations.
For features - first of all, I needed to encode categorical variables. For this I used basic one-hot encoding for some models, but also so-called lexical encoding, when value of encoded category is produced from its name (A becomes 0, B - 1, Z - 26, AA - 27, and so on).
I tried to find some meaningful features, but had no success at it. Also there were some kernels which provided insights into the nature of some variables, and tried to de-mangle them but I couldn’t get any improvement from it. So, I switched to using general automated methods.
The first of such methods was, of course, SVD, which I’ve applied to numerical variables and one-hot encoded categorical features. It helped to improve some high-variance models, like FM and NN.
Second, and more complex, was clustering the data and creating a new set of features based on the distance to cluster centers (i.e., applying RBF to them) - it helped to create a bunch of unsupervised non-linear features, which helped to improve most of my models.
And third, the last trick I used was forming categorical interaction features, applying lexical encoding to them. These combinations may be easily extracted from XGBoost models by just trying the most important categorical features, or better, analysing the model dump with the excellent Xgbfi tool.
First-level models
Based on these features, I built a lot of different models which I evaluated using the usual k-fold cross-validation.
First of all, there was linear regression, which gave me about 1237.43406 CV / 1223.28163 LB score, which is not very much of course, but provides some baseline. But after adding cluster features to it, it became 1202.70592 CV / 1189.64998 LB, which is much better for such a simple model.
Then, I tried scikit-learn RandomForestRegressor and ExtraTreesRegressor models, of which random forest was best, giving 1199.82233 CV / 1176.44433 LB after some tuning, and improved to 1186.23675 CV / 1166.85340 LB after adding categorical feature combinations. One problem with this model was that however scikit-learn supports MAE loss, it’s very slow and impossible to use, so I needed to use basic MSE, which has some bias in this competition.
The best model of scikit-learn which helped me was GradientBoostingRegressor, which was able to directly optimize MAE loss and gave me 1151.11060 CV / 1126.30971 LB
I also tried LibFM model, which gave me 1196.11333 CV / 1155.68632 LB in a basic version and 1177.69251 CV / 1150.37290 LB after adding cluster features to it.
But the main workhorses of this competitions were, of course, XGBoost and neural net models:
In the beginning, my XGBoost models provided something about 1133.00048 CV / 1112.86570 LB, but then I’ve applied some tricks which improved it to 1122.64977 CV / 1105.43686 LB:
- Averaging multiple runs of XGBoost with different seeds - it helps to reduce model variance;
- Adding categorical combination features;
- Modifying objective function to be closer to MAE;
- Tuning model parameters - I didn’t have much experience with it before, so this thread in Kaggle forums helped me a lot.
Custom objective function for XGBoost.
The other model that provided great results was neural net, implemented using the Keras library. I used basic multi-layer perceptron with 3 hidden layers which gave me somewhat about 1134.92794 CV / 1116.44915 LB in initial versions and improved to 1130.29286 CV / 1110.69527 LB after tuning and applying some tricks:
- Averaging multiple runs, again;
- Applying exponential moving average to weights of single network, using this implementation;
- Adding SVD and cluster features;
- Adding batch normalization and dropout;
Model tuning
In this competition, model hyperparameter tuning was very important, so I’ve contributed a lot of time in it. There are three main approaches here:
- Manual tuning, which works good when you have some intuition about parameter behaviour and may estimate model performance before its training completes by per-epoch validation scores;
- Uninformed parameter search - using GridSearchCV or RandomizedSearch from sklearn package, or similar - most simple of all;
- Informed search using HyperOpt or BayesOptimization or similar package - it tries to fit some model to scores of different parameter sets and selects the most promising point for each next try - so it usually finds the optimum a lot faster than uninformed search.
I used manual tuning for XGBoost and NN models which provide per-epoch validation scores and bayes optimization package for the others.
Second level
After getting a lot of models, I combined them in the second level, training new models on out-of-fold predictions:
- Linear regression, which gave me 1118.45564 CV / 1113.08059 LB score
- XGBoost - 1118.16984 CV / 1100.50998 LB
- Neural net - 1116.40752 CV / 1098.91721 LB (it was enough to get top-16 in public, and top-8 in private)
- Gradient boosting - 1117.41247 CV / 1099.60251 LB
I haven’t had much experience with stacking before and so I was really impressed by these results, but wanted to get even more.
So, the first thing I did was correct the bias of some stacked models - linear regression and XGBoost optimized some objective which was not equal to the objective of the competition, which resulted in overestimating low values and underestimating high ones. This bias is really small, but the competition was very close, so every digit counted.
This bias can be seen on the next figure where logs of XGBoost predictions are plotted against target logs with and a median regression line. If it was unbiased, the median regression should be the same as diagonal, but it's not (offset is most visible where red arrows are).
XGBoost bias.
I took XGBoost predictions to some small power p (around 1.03), normalized to preserve its median, and it improved my score to 1117.35084 CV / 1099.63060 LB.
Not bad, but maybe I can get even more by combining predictions of these models?
Third level
So, I built a third level. As each new stacking level becomes more and more unstable, I needed something really simple here, which may optimize the competition’s metric directly. So, I chose to use median regression from the statsmodels package.
The main problem of this approach was lack of regularization, so it wasn’t very stable and had a lot of noise. To fight it I applied some tricks:
- Training model on many subsamples and averaging predictions;
- Reducing input dimensionality - grouping similar models of previous layers and using group averages as features;
- Averaging best 10 submissions for a final one.
It allowed me to get 1098.07061 in public LB, and 1110.01364 in private, which corresponds to second place.
Final pipeline.
Lessons learned
So, this competitions helped me a lot, mainly in two main areas in which I lacked experience in before - model hyperparameter tuning (especially for XGBoost, where I got good hyperparameter intuition) and stacking, which I underestimated before.
Also, I tried a lot of different models and got intuition about how they work with different target, feature transformations, and so on.
Bio
Alexey Noskov is a Ruby and Scala developer at Evil Martians.
