[{"data":1,"prerenderedAt":211},["ShallowReactive",2],{"note-sofascore-data-traps-zh":3,"titles-zh":206},{"note":4,"project":136},{"id":5,"title":6,"body":7,"date":118,"description":119,"draft":120,"extension":121,"kind":122,"meta":123,"navigation":124,"path":125,"pinned":120,"project":126,"seo":127,"stem":129,"summary":130,"tags":131,"__hash__":135,"translated":124},"notes\u002Fnotes\u002Fsofascore-data-traps.md","一个赛季的 Sofascore 数据，以及它在哪里说谎",{"type":8,"value":9,"toc":111},"minimark",[10,14,17,21,29,32,35,70,73,83,86,89,100],[11,12,13],"p",{},"我的毕业论文需要 2025 年巴西甲级联赛的全部球员评分。Sofascore 有这些数据，就在它自己网站所用的内部 API 后面。由于这不是官方的公开 API，爬虫被刻意做得很\"无聊\"：每 1.2 到 2.4 秒发一次请求，失败时带退避重试，每一轮结束后保存断点，崩溃后可以接着跑而不必从头开始，并且每一行数据在写入 CSV 之前都要经过 Pydantic 校验。",[11,15,16],{},"最终从全部 380 场比赛中抓取了 11,977 条球员单场记录，比赛时间从 2025 年 3 月 29 日到 12 月 7 日。采集是最简单的部分。数据里有三个陷阱，而且没有一个会报错。",[18,19,20],"h2",{"id":20},"时区错误的日期",[11,22,23,24,28],{},"爬虫用 UTC 存储比赛日期。巴西的夜场比赛在 21:00 或更晚开球，按 UTC 已经是第二天了，所以 380 场比赛中有 67 场的日期是错的。修正方法是根据开球时间戳在 ",[25,26,27],"code",{},"America\u002FSao_Paulo"," 时区下重新计算日期，清洗步骤和爬虫本身都做了修改。",[18,30,31],{"id":31},"三种缺失",[11,33,34],{},"一个空单元格可能代表三种不同的含义，每种都有各自的处理规则：",[36,37,38,50,56],"ul",{},[39,40,41,45,46,49],"li",{},[42,43,44],"strong",{},"不适用。"," 比如中锋的扑救数。这些字段对非门将球员根本不会出现，所以填 ",[25,47,48],{},"-1","，这是任何真实计数都不可能取到的值。",[39,51,52,55],{},[42,53,54],{},"未测量。"," 体能数据（跑动距离、冲刺次数、最高速度）只在 380 场中的 349 场有，带球推进数据只在 83 场有。这些保持缺失。如果填成零，就等于声称球员没有跑动。",[39,57,58,61,62,65,66,69],{},[42,59,60],{},"为零而被省略。"," Sofascore 不会发送值为 0 的数据。在原始数据中，",[25,63,64],{},"goals"," 有 11,124 行为空，",[25,67,68],{},"expected_goals"," 有 6,314 行为空，恰好都是球员没有射门的那些行。这些填为 0。",[11,71,72],{},"在体能和带球推进这两组数据中，规则按球员判断：如果某球员有该组中的任意一项指标，缺失的就是被省略的零值；如果一项都没有，说明这组数据没有为他测量。",[11,74,75,76,79,80,82],{},"一个边界情况：门将的 ",[25,77,78],{},"goals_prevented"," 可能为负（从 -3.79 到 3.12），所以在这里哨兵值 ",[25,81,48],{}," 会和真实值冲突。这时用位置列来区分。",[18,84,85],{"id":85},"打乱顺序的轮次",[11,87,88],{},"给一个赛季排序，最直观的方法是按轮次编号。但这是错的。轮次会因为洲际赛事而改期，第 34 轮的大部分比赛直到 11 月才踢，而第 35 到 37 轮早在 10 月就已经踢完了。整个赛季中，72,010 对可能的比赛组合里，有 1,795 对（2.49%）的轮次编号和实际开球时间相矛盾。",[11,90,91,92,95,96,99],{},"一旦要计算球员的历史数据，这就不再是冷知识了。按轮次而不是开球时间排序，会改变 686 名踢过两场或以上的球员中 335 人（48.8%）的比赛顺序。对于一个只能看到过去比赛的模型来说，这就是数据泄漏：一场未来的比赛混进了过去某场比赛的历史里。之后的所有步骤都按 ",[25,93,94],{},"kickoff_timestamp"," 排序，从不按 ",[25,97,98],{},"round"," 排序。",[11,101,102,103,110],{},"清洗之后，删除了 163 条没有评分的出场记录，它们的出场时间都在 7 分钟或以下。最终剩下 732 名球员的 11,814 行数据，也就是 ",[104,105,109],"a",{"className":106,"href":108},[107],"wikilink","\u002Fzh\u002Fnotes\u002Fr2-too-good","当 R² 好得过头"," 中模型所用的训练数据。",{"title":112,"searchDepth":113,"depth":113,"links":114},"",2,[115,116,117],{"id":20,"depth":113,"text":20},{"id":31,"depth":113,"text":31},{"id":85,"depth":113,"text":85},"2026-10-22","My thesis needed every player rating from the 2025 Brazilian Série A. Sofascore has them, behind the same internal API its website uses. It isn't an official public API, so the crawler was built to be boring: one request every 1.2 to 2.4 seconds, retries with backoff, a checkpoint after every round so a crash resumes instead of starting over, and Pydantic validation on every row before it reaches the CSV.",false,"md","article",{},true,"\u002Fnotes\u002Fsofascore-data-traps","sofascore-rating-prediction",{"title":128,"description":119},"A season of Sofascore data, and where it lies","notes\u002Fsofascore-data-traps","抓取 2025 年巴西甲级联赛的全部球员评分，以及数据里的三个陷阱：时区错误的日期、从不发送的零值，以及打乱顺序的轮次。",[132,133,134],"python","scraping","data","eXQcuImFjV98ThxZr0pIfb9o9ar0a4y7Y8iJLRS93iQ",{"id":137,"title":138,"body":139,"cover":190,"date":191,"demo":190,"description":192,"draft":120,"extension":121,"featured":124,"meta":193,"navigation":124,"order":194,"path":195,"repo":196,"seo":197,"status":199,"stem":200,"summary":201,"tags":202,"__hash__":205,"translated":124},"projects\u002Fprojects\u002Fsofascore-rating-prediction.md","Sofascore 球员评分预测",{"type":8,"value":140,"toc":185},[141,144,147,150,170,173,176,179,182],[11,142,143],{},"我在 FATEC Sorocaba 的毕业论文，指导老师是 Maria das Graças J. M. Tomazela 教授。研究问题只有一个：按位置（门将、后卫、中场、前锋）分别训练的随机森林模型，能否比单一的通用模型以更小的误差在赛前预测球员的 Sofascore 评分？",[18,145,146],{"id":146},"流程",[11,148,149],{},"分三个阶段，每个阶段只从磁盘读取上一阶段的输出，因此任何一个阶段都可以单独重跑。",[36,151,152,158,164],{},[39,153,154,157],{},[42,155,156],{},"采集。"," 一个\"有礼貌\"的 Sofascore 内部 API 爬虫：限速、带退避的重试、断点续爬，并对每一行数据做 Pydantic 校验。共采集 11,977 条球员单场记录，覆盖 2025 年巴西甲级联赛全部 380 场比赛。",[39,159,160,163],{},[42,161,162],{},"清洗。"," 每个步骤一个函数。有意思的是，一个空单元格可能代表三种不同的含义，每种都有各自的处理规则。",[39,165,166,169],{},[42,167,168],{},"建模。"," 只使用赛前特征：主客场、对手实力，以及球员截至被预测比赛之前的历史表现。在最后 80 场比赛上做前向滚动验证，以整场比赛为单位做 bootstrap 置信区间，并计算置换重要性。",[18,171,172],{"id":172},"结果",[11,174,175],{},"按位置专门建模并没有降低误差。通用模型比\"按位置取平均\"的基线好 6%（MAE 0.374 对 0.398），而门将的评分在赛前几乎无法预测。一场比赛的评分，大部分是在比赛过程中才决定的。",[18,177,178],{"id":178},"笔记",[11,180,181],{},"按顺序讲完整个过程：数据里的陷阱，第一个好得不真实的模型，以及当模型只能看到过去时得出的结果。",[11,183,184],{},"技术栈：Python、httpx、tenacity、Pydantic v2、pandas、scikit-learn、pytest。",{"title":112,"searchDepth":113,"depth":113,"links":186},[187,188,189],{"id":146,"depth":113,"text":146},{"id":172,"depth":113,"text":172},{"id":178,"depth":113,"text":178},null,"2026-08-27","My undergraduate thesis at FATEC Sorocaba, supervised by Prof. Dr. Maria das Graças J. M. Tomazela. One question: can Random Forest models specialized by position (goalkeeper, defender, midfielder, forward) predict a player's Sofascore rating before the match with lower error than a single generic model?",{},3,"\u002Fprojects\u002Fsofascore-rating-prediction","https:\u002F\u002Fgithub.com\u002Falmeidaota\u002Fsofascore-rating-prediction",{"title":198,"description":192},"Sofascore rating prediction","active","projects\u002Fsofascore-rating-prediction","我的毕业论文。按位置分别训练的随机森林模型，能否比一个通用模型更准确地在赛前预测球员的 Sofascore 评分？答案是否定的。",[132,133,203,204],"ml","random-forest","IcTcWXpYn5Kpmy9fqNl7cIFEbUs7Jrm8SMXO46pBHt8",{"\u002Fnotes\u002Fva-longe-mas-va-acompanhado":207,"\u002Fnotes\u002Fpre-match-ratings":208,"\u002Fnotes\u002Fr2-too-good":109,"\u002Fnotes\u002Fsofascore-data-traps":6,"\u002Fprojects\u002Fargus-maximus":209,"\u002Fprojects\u002Frobbie":210,"\u002Fprojects\u002Fsofascore-rating-prediction":138},"走得远，就要结伴同行","一场比赛的评分，大部分无法预测","Argus Maximus","robbie",1793409193275]