Catatan episode
today we are taking a deeper look at of the secret behind LLM pretraining which is synthetic data pipelines with Joel Niklaus machine learning engineer at Hugging Face!joel an his team ran 90 controlled experiments and burned over a trillion tokens to figure out what actually makes good pretraining data and in huggingface fashion provided all of their findings/artifacts openly!
lots of things to learn from experiment like this and to be honest there is still so much more to do!
couple of useful links:
👉Joel Website: https://niklaus.ai👉 Twitter: https://x.com/joelniklaus👉 Paper : arXiv…
Situs: yacinemahdid.com

