The Llama 3 Herd of Models (Meta official technical report)
2024/07/31 · 公開情報から自動収集したものです。人による確認は行っていません。
この内容は原文の言語（英語）でのみ公開されています。
What happened
Meta's technical report on the Llama 3 herd of models states that Llama 3 was pre-trained on about 15 trillion multilingual tokens, compared with 1.8 trillion for Llama 2. The flagship 405-billion-parameter model was trained on 15.6 trillion tokens using 3.8 x 10^25 FLOPs, roughly 50 times the compute of the largest Llama 2 model. The paper discloses no specific dataset sources.
Why it may matter to you
A 15-trillion-token corpus is the scale of training data that determines whose work ends up inside one of the most widely used open models — and the report does not say which texts were used.
What is not yet known
The paper does not list the datasets or documents the tokens came from, so creators cannot check whether their work is in the corpus.
出典
- The Llama 3 Herd of Models — arXiv (primary)