[{"data":1,"prerenderedAt":209},["ShallowReactive",2],{"note-sofascore-data-traps-en":3,"titles-en":208},{"note":4,"project":137},{"id":5,"title":6,"body":7,"date":121,"description":13,"draft":122,"extension":123,"kind":124,"meta":125,"navigation":126,"path":127,"pinned":122,"project":128,"seo":129,"stem":130,"summary":131,"tags":132,"__hash__":136,"translated":126},"notes\u002Fnotes\u002Fsofascore-data-traps.md","A season of Sofascore data, and where it lies",{"type":8,"value":9,"toc":114},"minimark",[10,14,17,22,30,34,37,72,75,85,89,92,103],[11,12,13],"p",{},"My thesis needed every player rating from the 2025 Brazilian Série A. Sofascore has them, behind the same internal API its website uses. It isn't an official public API, so the crawler was built to be boring: one request every 1.2 to 2.4 seconds, retries with backoff, a checkpoint after every round so a crash resumes instead of starting over, and Pydantic validation on every row before it reaches the CSV.",[11,15,16],{},"That pulled 11,977 player-match records out of all 380 matches, played between March 29 and December 7, 2025. Collecting was the easy part. The data had three traps in it, and none of them raised an error.",[18,19,21],"h2",{"id":20},"dates-in-the-wrong-timezone","Dates in the wrong timezone",[11,23,24,25,29],{},"The crawler stored match dates in UTC. Brazilian night games kick off at 21:00 or later, which is already the next day in UTC, so 67 of the 380 matches had the wrong date. The fix was recomputing the date from the kickoff timestamp in ",[26,27,28],"code",{},"America\u002FSao_Paulo",", both in the cleaning step and in the crawler itself.",[18,31,33],{"id":32},"three-kinds-of-missing","Three kinds of missing",[11,35,36],{},"An empty cell meant three different things, and each one got its own rule:",[38,39,40,52,58],"ul",{},[41,42,43,47,48,51],"li",{},[44,45,46],"strong",{},"Doesn't apply."," Saves for a striker. Those keys never appear for outfield players, so they get ",[26,49,50],{},"-1",", a value no real count can take.",[41,53,54,57],{},[44,55,56],{},"Not measured."," Physical stats (distance, sprints, top speed) exist in only 349 of the 380 matches, ball carries in only 83. These stay missing. Filling them with zero would claim a player didn't run.",[41,59,60,63,64,67,68,71],{},[44,61,62],{},"Zero, omitted."," Sofascore doesn't send stats that are 0. In the raw data, ",[26,65,66],{},"goals"," was empty in 11,124 rows and ",[26,69,70],{},"expected_goals"," in 6,314, exactly the rows where the player took no shot. These become 0.",[11,73,74],{},"Inside the physical and carries groups the rule is per player: if a player has any metric of the group, the missing ones are omitted zeros. If they have none, the group wasn't measured for them.",[11,76,77,78,81,82,84],{},"One edge case: ",[26,79,80],{},"goals_prevented"," can be negative for goalkeepers (from -3.79 to 3.12), so the ",[26,83,50],{}," sentinel collides with real values there. The position column breaks the tie.",[18,86,88],{"id":87},"rounds-out-of-order","Rounds out of order",[11,90,91],{},"The obvious way to sort a season is by round number. It's wrong. Rounds get rescheduled around continental competitions, and most of round 34 was only played in November, after rounds 35 to 37 had been played in October. Across the season, 1,795 of the 72,010 possible pairs of matches (2.49%) have round numbers that contradict their kickoff times.",[11,93,94,95,98,99,102],{},"That stops being trivia as soon as you compute a player's history. Sorting by round instead of kickoff changes the order of matches for 335 of the 686 players (48.8%) with two or more games. For a model that is only allowed to look at past matches, that is leakage: a future game sneaking into the history of a past one. Everything downstream sorts by ",[26,96,97],{},"kickoff_timestamp",", never by ",[26,100,101],{},"round",".",[11,104,105,106,113],{},"After cleaning, 163 appearances without a rating were dropped, all of them 7 minutes long or shorter. That left 11,814 rows from 732 players, which is what the models in ",[107,108,112],"a",{"className":109,"href":111},[110],"wikilink","\u002Fnotes\u002Fr2-too-good","When R² is too good"," were trained on.",{"title":115,"searchDepth":116,"depth":116,"links":117},"",2,[118,119,120],{"id":20,"depth":116,"text":21},{"id":32,"depth":116,"text":33},{"id":87,"depth":116,"text":88},"2026-10-22",false,"md","article",{},true,"\u002Fnotes\u002Fsofascore-data-traps","sofascore-rating-prediction",{"title":6,"description":13},"notes\u002Fsofascore-data-traps","Scraping every player rating from the 2025 Brasileirão, and the three traps in the data: dates in the wrong timezone, zeros that never get sent, and rounds played out of order.",[133,134,135],"python","scraping","data","eXQcuImFjV98ThxZr0pIfb9o9ar0a4y7Y8iJLRS93iQ",{"id":138,"title":139,"body":140,"cover":194,"date":195,"demo":194,"description":144,"draft":122,"extension":123,"featured":126,"meta":196,"navigation":126,"order":197,"path":198,"repo":199,"seo":200,"status":201,"stem":202,"summary":203,"tags":204,"__hash__":207,"translated":126},"projects\u002Fprojects\u002Fsofascore-rating-prediction.md","Sofascore rating prediction",{"type":8,"value":141,"toc":189},[142,145,149,152,172,176,179,183,186],[11,143,144],{},"My undergraduate thesis at FATEC Sorocaba, supervised by Prof. Dr. Maria das Graças J. M. Tomazela. One question: can Random Forest models specialized by position (goalkeeper, defender, midfielder, forward) predict a player's Sofascore rating before the match with lower error than a single generic model?",[18,146,148],{"id":147},"pipeline","Pipeline",[11,150,151],{},"Three stages, each one reading only the previous stage's output from disk, so any of them can be rerun alone.",[38,153,154,160,166],{},[41,155,156,159],{},[44,157,158],{},"Collection."," A polite crawler for Sofascore's internal API: rate limiting, retries with backoff, checkpoint\u002Fresume and Pydantic validation on every row. 11,977 player-match records from all 380 matches of the 2025 Brazilian Série A.",[41,161,162,165],{},[44,163,164],{},"Cleaning."," One function per step. The interesting part is that an empty cell can mean three different things, each with its own rule.",[41,167,168,171],{},[44,169,170],{},"Modelling."," Pre-match features only: home\u002Faway, opponent strength and the player's own history up to the match being predicted. Walk-forward validation over the last 80 matches, bootstrap confidence intervals over whole matches, permutation importance.",[18,173,175],{"id":174},"result","Result",[11,177,178],{},"Specialization didn't reduce the error. The generic model beat a position-average baseline by 6% (MAE 0.374 vs 0.398), and predicting goalkeepers turned out to be hopeless before kickoff. Most of a match rating is decided during the match.",[18,180,182],{"id":181},"notes","Notes",[11,184,185],{},"The story in order: the data traps, the first model that was too good to be true, and what came out once the model could only see the past.",[11,187,188],{},"Stack: Python, httpx, tenacity, Pydantic v2, pandas, scikit-learn, pytest.",{"title":115,"searchDepth":116,"depth":116,"links":190},[191,192,193],{"id":147,"depth":116,"text":148},{"id":174,"depth":116,"text":175},{"id":181,"depth":116,"text":182},null,"2026-08-27",{},3,"\u002Fprojects\u002Fsofascore-rating-prediction","https:\u002F\u002Fgithub.com\u002Falmeidaota\u002Fsofascore-rating-prediction",{"title":139,"description":144},"active","projects\u002Fsofascore-rating-prediction","My undergraduate thesis. Can Random Forest models specialized by position predict a player's Sofascore rating before kickoff better than one generic model? The answer was no.",[133,134,205,206],"ml","random-forest","IcTcWXpYn5Kpmy9fqNl7cIFEbUs7Jrm8SMXO46pBHt8",{},1793409193593]