The Llama 3 Herd of Models (Meta official technical report)
2024. 7. 31. · 공개 출처에서 자동으로 수집했습니다. 사람이 검토하지 않았습니다.
이 항목은 원문 언어(영어)로만 제공됩니다.
What happened
Meta's technical report on the Llama 3 herd of models states that Llama 3 was pre-trained on about 15 trillion multilingual tokens, compared with 1.8 trillion for Llama 2. The flagship 405-billion-parameter model was trained on 15.6 trillion tokens using 3.8 x 10^25 FLOPs, roughly 50 times the compute of the largest Llama 2 model. The paper discloses no specific dataset sources.
Why it may matter to you
A 15-trillion-token corpus is the scale of training data that determines whose work ends up inside one of the most widely used open models — and the report does not say which texts were used.
What is not yet known
The paper does not list the datasets or documents the tokens came from, so creators cannot check whether their work is in the corpus.
출처
- The Llama 3 Herd of Models — arXiv (primary)