The Llama 3 Herd of Models (Meta official technical report)
2024年7月31日 · 自動從公開來源蒐集，未經人工審核。
這篇內容目前只有 英文 原文版本。
What happened
Meta's technical report on the Llama 3 herd of models states that Llama 3 was pre-trained on about 15 trillion multilingual tokens, compared with 1.8 trillion for Llama 2. The flagship 405-billion-parameter model was trained on 15.6 trillion tokens using 3.8 x 10^25 FLOPs, roughly 50 times the compute of the largest Llama 2 model. The paper discloses no specific dataset sources.
Why it may matter to you
A 15-trillion-token corpus is the scale of training data that determines whose work ends up inside one of the most widely used open models — and the report does not say which texts were used.
What is not yet known
The paper does not list the datasets or documents the tokens came from, so creators cannot check whether their work is in the corpus.
來源
- The Llama 3 Herd of Models — arXiv (primary)