Notatki do odcinka
Should you recalibrate CTR models on fresh data? Rui Liu's Avazu study: a 6-hour Platt/isotonic refit made calibration worse in about a third of applications.
Key takeaways:
• Random splits flatter calibration. On Avazu (40.4M impressions; LR, DeepFM, DCN-v2), a random split reported LR's ECE at 55–62% of its value on the nearest future day in every seed. For the deep models the ratio was seed-unstable, and no model's ECE grew steadily with distance from training.
• Recalibration can backfire. Platt or isotonic refits on the preceding 6 hours removed up to 61% of ECE in badly miscalibrated…
Strona: rss.com

