Get through the pile.
Paste several links at once. Yank finds the five things worth knowing in each, with their sources, then read them in turn, or play them through: videos and podcasts in their own voices, everything else read aloud by an AI voice.
1 source · 5 lines · about 1 min to play
1 sources · 1 min in all
Videos and podcasts play in their own players, from two seconds before each line. Articles and documents are read aloud by an AI voice from ElevenLabs, reading only the words already on each yank's page, or by your device's own voice if you choose it. It keeps playing if you go to another page.
BERT: Bidirectional Pre-training Breaks the NLP Leaderboard
The landmark Google AI paper that introduced BERT, demonstrating how masking tokens and conditioning on both left and right context simultaneously produced a single pre-trained model that shattered eleven NLP benchmarks at once.
- BERT is the first fine-tuning based representation model that achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks, outperforming many task-specific architectures.
- Bidirectional conditioning would allow each word to indirectly 'see itself', and the model could trivially predict the target word in a multi-layered context — so BERT instead masks 15% of tokens at random and predicts only those.
- This is the first work to demonstrate convincingly that scaling to extreme model sizes also leads to large improvements on very small scale tasks, provided that the model has been sufficiently pre-trained.
- BERT LARGE obtains a GLUE score of 80.5 versus OpenAI GPT's 72.8 — an architecture otherwise nearly identical except for attention direction.
- The Transformer encoder does not know which words it will be asked to predict or which have been replaced by random words, so it is forced to keep a distributional contextual representation of every input token.