The Llama 3 Herd of Models (Meta official technical report)
31 जुल॰ 2024 · सार्वजनिक स्रोतों से स्वचालित रूप से एकत्र किया गया। किसी व्यक्ति द्वारा समीक्षित नहीं।
यह सामग्री केवल अपनी मूल भाषा (अंग्रेज़ी) में उपलब्ध है।
What happened
Meta's technical report on the Llama 3 herd of models states that Llama 3 was pre-trained on about 15 trillion multilingual tokens, compared with 1.8 trillion for Llama 2. The flagship 405-billion-parameter model was trained on 15.6 trillion tokens using 3.8 x 10^25 FLOPs, roughly 50 times the compute of the largest Llama 2 model. The paper discloses no specific dataset sources.
Why it may matter to you
A 15-trillion-token corpus is the scale of training data that determines whose work ends up inside one of the most widely used open models — and the report does not say which texts were used.
What is not yet known
The paper does not list the datasets or documents the tokens came from, so creators cannot check whether their work is in the corpus.
स्रोत
- The Llama 3 Herd of Models — arXiv (primary)