The thesis question: can Random Forest models specialized by position (goalkeeper, defender, midfielder, forward) predict a player's Sofascore rating better than one generic model?
The first version fed each model the player's stats from the match itself: shots, passes, duels, tackles, touches, 63 columns in total. The validation was clean. Train on rounds 1 to 30, test on 31 to 38, tune hyperparameters with GroupKFold grouped by match, so players from the same game never end up on both sides of a fold. A baseline that just predicts the position's average rating set the floor.
| Model | MAE | R² |
|---|---|---|
| Position average | 0.391 | |
| Generic | 0.203 | 0.72 |
| Specialists | 0.207 | 0.72 |
Forwards reached R² 0.80. Cutting the baseline error in half reads great in a thesis. It also didn't add up: one match's worth of stats shouldn't explain that much of anything in football.
Not leakage. Circularity.
The first suspect was train/test leakage. There wasn't any. The split and the folds kept every match whole, and no test row touched training.
The problem was the target. The Sofascore rating comes out of a proprietary formula fed by stats that look a lot like those 63 features: goals, passes, duels, tackles. The Random Forest wasn't learning what a good performance is. It was approximating the formula that produced the number. The high R² measured how well the model imitates Sofascore's algorithm, and nothing else.
That experiment still answered a narrower question (with the match stats in hand, specialists only won for goalkeepers, the most statistically distinct position). But it couldn't answer the one that matters: can you know a rating before the game?
The redesign
The second experiment bans every stat from the match being predicted. Each feature has to exist before kickoff:
- home or away;
- opponent strength: goals scored and conceded by the opponent in earlier matches;
- the player's own history: season averages of 61 stats, plus season and last-5 averages of rating and minutes, all computed only from matches strictly before this one.
"Strictly before" is why the round ordering in A season of Sofascore data, and where it lies stopped being a detail. Players with no history yet (14.9% of them play 3 games or fewer all season) get NaN, which scikit-learn's Random Forest handles natively since 1.4. A count of prior matches goes in alongside, so the model can learn to trust 20 games of history more than 1.
Validation moved to walk-forward: train on everything before a batch of 10 matches, predict that batch, add it to the training set, move on, across the last 80 matches of the season. Hyperparameter search uses time-ordered folds inside the training window, always validating on matches that came after the training ones.
R² dropped to 0.10. That's the expected result, not a regression. Explaining a rating that already happened, from stats close to its own formula, is one task. Predicting it before the game is a different and much harder one. Most of a match rating can't be predicted covers what came out.