This is an automated archive made by the Lemmit Bot.

The original was posted on /r/machinelearning by /u/datashri on 2025-06-21 05:46:46+00:00.

I’m reading through the Qwen2 paper.

Something escapes my limited comprehension -

Section 3.1

… the pre-training data was expanded from 3 trillion tokens in Qwen1.5 (Qwen Team, 2024a) to 7 trillion tokens. An attempt to further relax the quality threshold resulted in a 12 trillion token dataset. However, the model trained on this dataset did not show a significant performance improvement over the 7 trillion token model. It is suspected that increasing the volume of data does not necessarily benefit model pre-training.

So higher quality smaller dataset is better. Got it.

All Qwen2 dense models, excluding Qwen2-0.5B, were pre-trained on this large-scale dataset of over 7 trillion tokens. Qwen2-0.5B were pre-trained using the 12 trillion token dataset.

How is it conceivable to train that tiny model on the humongous but lower quality dataset?? My modest intellect feels borderline abused.

Appreciate any tips to guide my understanding.