Qwen3 Technical Report — Alibaba's Qwen team discloses 36 trillion-token pre-training corpus
2025年5月14日 · 自動從公開來源蒐集，未經人工審核。
這篇內容目前只有 英文 原文版本。
What happened
Alibaba's Qwen team published the Qwen3 technical report on 14 May 2025. Qwen3 was pre-trained on about 36 trillion tokens across 119 languages and dialects, twice as many tokens as Qwen2.5 (about 18 trillion) and up from 29 languages. To expand the corpus, the team used Qwen2.5-VL to extract text from extensive PDF documents (trillions of tokens in total) and generated trillions of synthetic tokens with Qwen2.5-family models. The paper discloses no specific source-dataset list.
Why it may matter to you
A 36-trillion-token corpus built partly from PDF documents and synthetic text is the scale of training data that determines whose work ends up inside a widely used open model — and the report does not say which documents were used.
What is not yet known
The paper does not list the datasets or documents the tokens came from, so creators cannot check whether their work is in the corpus.
來源
- Qwen3 Technical Report — arXiv (primary)