Qwen3 Technical Report — Alibaba's Qwen team discloses 36 trillion-token pre-training corpus
2025. 5. 14. · 공개 출처에서 자동으로 수집했습니다. 사람이 검토하지 않았습니다.
이 항목은 원문 언어(영어)로만 제공됩니다.
What happened
Alibaba's Qwen team published the Qwen3 technical report on 14 May 2025. Qwen3 was pre-trained on about 36 trillion tokens across 119 languages and dialects, twice as many tokens as Qwen2.5 (about 18 trillion) and up from 29 languages. To expand the corpus, the team used Qwen2.5-VL to extract text from extensive PDF documents (trillions of tokens in total) and generated trillions of synthetic tokens with Qwen2.5-family models. The paper discloses no specific source-dataset list.
Why it may matter to you
A 36-trillion-token corpus built partly from PDF documents and synthetic text is the scale of training data that determines whose work ends up inside a widely used open model — and the report does not say which documents were used.
What is not yet known
The paper does not list the datasets or documents the tokens came from, so creators cannot check whether their work is in the corpus.
출처
- Qwen3 Technical Report — arXiv (primary)