Dario> We are virtuous!
(steals the internet)
China> We stole the internet frfr
bebuX
View on Xhttp://z.ai/ chief scientist Jie Tang just revealed the training pipeline behind the upcoming GLM-5.5 (32B, 122B, and 256B) during a lecture at Tsinghua University today.
Instead of relying on public datasets, their pretraining pipeline pulls from six separate internal engines:
- Web: 1 trillion pages crawled at ~10B pages/day with PB-scale template clustering
- Documents: 2 billion scientific PDFs parsed through GLM-OCR (4x faster with speculative decoding)
- Books: 50 million books with layout reordering and garbled-text filtering
- Code: 500 million repositories turned into executable tasks with real tests and issue descriptions
- Video: 10 billion videos for long-sequence world knowledge
- Audio: 300 million hours rewritten via ASR into trillions of tokens
1
0 comments