Imagine you’re preparing for an exam. You sit a first practice test and look at the results: you’re good at geometry but get lots of probability questions wrong. The smart move is to spend next week on probability, not geometry. After the second test you look again: probability has improved, now word problems are the weak spot. So next week you work on word problems.
Each round, you focus on whatever is still missing from the round before. That’s exactly the idea behind boosting.
Trees That Work In Turn
In our Random Forest post we explained how hundreds of trees are trained at the same time, independently, and then vote. In boosting, trees are trained one after another:
- The first tree makes a simple prediction. It’s right on some examples and wrong on others.
- The second tree looks at where and by how much the first one was wrong, and tries to fix only that error.
- The third tree focuses on whatever error is left after the first two.
And so on. Each tree on its own is a weak predictor, but together, each filling in the gaps left by the one before, they form a very strong model. Like editors reading a text in turn: each one catches the mistakes the previous one missed.
Where Does The “X” Come From?
XGBoost (eXtreme Gradient Boosting) is a particularly fast and careful implementation of this idea. It does two things well:
- A brake: it penalises trees for becoming too complex and memorising the training data. In other words: “fix the error, but don’t overdo it.”
- Speed: it was designed to run its calculations in parallel, so it trains quickly even on large datasets.
Together, these made XGBoost one of the most successful methods for tabular data, data made of rows and columns. For years it was one of the tools most often used by winning teams in data science competitions.
Why Does It Matter To Us?
The features we extract from EEG recordings, such as the power of each frequency band at each electrode, form exactly this kind of table: each row a trial, each column a feature. Because XGBoost performs well and is relatively fast on such tables, it’s one of the methods we often turn to in studies like emotion and liking classification.
References
- Chen, T., Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. DOI: 10.1145/2939672.2939785