Techno Optimist on Instagram: "Zuck and Wang on how to pre-train an AI model: Wang: “In the pre-training stage, everyone who’s training frontier models basically is trying to get as much data as poss
A rare candid exchange between two AI insiders on what it actually feels like — and costs — to pre-train a frontier model at scale.
1 / 5at 0:08 · Yank's paraphrase
Pre-training is full of researchers who chew glass and stare out into the abyss.
AI Summary
In a conversation between Zuck and Wang about pre-training AI models, Wang describes pre-training as the highest-stakes research area, involving emotional hardship and scientific uncertainty due to limited experimental samples. He explains that frontier model developers aim to gather as much data as possible, with intentional focus on desired domains, while compute and data must scale together. A key challenge is that as models grow bigger, data must grow with them — yet the internet is not growing fast enough to match that demand. The conversation closes by flagging post-training as a now-exciting next frontier.
Yank Summary
What matters, in Yank’s words, each linked to where the source says it.
Wang describes pre-training as the highest-stakes research area, with developers investing a huge percentage of their compute into it. 0:00
Wang says everyone training frontier models tries to get as much data as possible, with a bent toward the specific corpus matching the model's intended domain. 0:45
Pre-training is full of researchers who chew glass and stare out into the abyss.
A viral quote frames pre-training as emotionally and psychologically brutal work, not just technical labor.
In context
A lot of the art of pre -training, I mean, pre -training is, there's this one rune tweet where it's like, pre -training is full of the researchers who like, chew glass and stare out into the distance and into the abyss or whatnot. But I think there's a truth to that because it is like, it is both the highest stakes of the research areas because you, know, we invested a huge percentage of our compute into training Muse Wonder, into pre -training Muse Wonder. And it was very high stakes. I think probably both and it was very high stakes. I think that's
Pre-training is full of the researchers who like, chew glass and stare out into the distance and into the abyss.
Receipt
All the seven stages of grief happen while you're pre-training a model.
Zuck maps the full arc of emotional suffering researchers endure before a model's outcome is known.
In context
high stakes. I think probably both and it was very high stakes. I think that's probably both of us at various points in that process. You know, you went through troughs of sadness, troughs of uncertainty. You know, all the seven stages of grief, you know, happen while you're while you're pre -training a model. And it is the most fundamentally scientific in an interesting way because, you know, you just don't have very many samples. So like your ability to infer exactly what will happen scientifically really depends on your ability to sort of like, you know, big brain reason about everything
All the seven stages of grief you know, happen while you're while you're pre -training pre-training a model.
Receipt
Scaling bigger models isn't just about flops and compute — you need to scale data with it, and if the internet isn't growing that quickly, the question becomes: where does the data come from?
The data bottleneck is identified as the binding constraint on scaling, not raw compute power.
In context
positive transfer. But then, depending on how it's set up, you can have some negative transfer. So I don't know. There are all these interesting questions. But in general, one of the bigger challenges of scaling bigger models is it's not just about how many flops and how much compute. You want to be able to scale data with that. I mean, this is like part of what when the original kind of scaling laws for pre -training were written. I mean, this is sort of what came out of that is that so if the Internet isn't growing that quickly and you want to
Scaling bigger models is it's not isn't just about how many flops and how much compute — you want to be able need to scale data with that. I mean, this is like part of what when the original kind of scaling laws for pre -training were written. I mean, this is sort of what came out of that is that so it, and if the internet isn't growing that quickly, the question becomes: where does the data come from?
Receipt
Your ability to infer exactly what will happen scientifically really depends on your ability to big-brain reason about everything that's happening.
Low sample counts force researchers to rely on first-principles reasoning rather than empirical iteration.
In context
troughs of sadness, troughs of uncertainty. You know, all the seven stages of grief, you know, happen while you're while you're pre -training a model. And it is the most fundamentally scientific in an interesting way because, you know, you just don't have very many samples. So like your ability to infer exactly what will happen scientifically really depends on your ability to sort of like, you know, big brain reason about everything that's happening. in this process. Okay, so in the pre -training stage, everyone who's training
Your ability to infer exactly what will happen scientifically really depends on your ability to sort of like, you know, big brain big-brain reason about everything that's happening.
Receipt
You still want as much intentionality about where ultimately you want your model to be — but it just benefits from a lot more data.
Even massive pre-training runs reward deliberate curation toward a model's intended downstream purpose.
In context
-train with maybe a bent towards if you want use Wonder to excel at legal and finance and tax questions and things like this, then you want to have as much of that specific corpus as possible in the pre -train. You still want as much intentionality about where ultimately you want your model to be, but again it just you know benefits from a lot more data. Yeah. And this is... sort of the massive data massive compute stage do you hit some saturation in the data you put in the tree train or ultimately is it just more data is
You still want as much intentionality about where ultimately you want your model to be — but again it just you know benefits from a lot more data.
Receipt
Keep going with this source
Create something
Turn this source into a post, a thread or a newsletter, every line checked against it.
See what you can create →Ask a question
Ask about anything in this video. We’ll try to provide an answer.