The Llama 3 Herd of Models (Meta official technical report)
31 juil. 2024 · Collecté automatiquement à partir de sources publiques. Non vérifié par une personne.
Ce contenu n'est disponible que dans sa langue d'origine (anglais).
What happened
Meta's technical report on the Llama 3 herd of models states that Llama 3 was pre-trained on about 15 trillion multilingual tokens, compared with 1.8 trillion for Llama 2. The flagship 405-billion-parameter model was trained on 15.6 trillion tokens using 3.8 x 10^25 FLOPs, roughly 50 times the compute of the largest Llama 2 model. The paper discloses no specific dataset sources.
Why it may matter to you
A 15-trillion-token corpus is the scale of training data that determines whose work ends up inside one of the most widely used open models — and the report does not say which texts were used.
What is not yet known
The paper does not list the datasets or documents the tokens came from, so creators cannot check whether their work is in the corpus.
Sources
- The Llama 3 Herd of Models — arXiv (primary)