4-Bit Training for Billion-Parameter LLMs? Yes, Really.
The video discusses training large language models using FP4 quantization to reduce computational costs while maintaining performance.
s1: Simple test-time scaling: Just “wait…” + 1,000 training examples? | PAPER EXPLAINED
A thousand well-chosen examples can train LLMs to output reasoning chains effectively, using a simple test-time trick to enhance performance.
Only self-supervision
The video explains two main types of self-supervised learning: mask language modeling and autoregressive language modeling.
Training large language models to reason in a continuous latent space – COCONUT Paper explained
COCONUT trains language models to reason in a continuous latent space, improving efficiency and flexibility.
Why do we need Tokenizers for LLMs?
The video discusses how to represent text as vectors for language models using tokenization to handle new words and typos.
REPA Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You ...
The video discusses a paper introducing the REPA loss term for diffusion models, which enhances their ability to learn general-purpose image representations by leveraging pretrained models like DINOv2, resulting in faster and more effective training.