🧠 Self-attention is the single most important idea in modern AI — and most tutorials get it wrong. In this video, you will see exactly how self-attention works: from the raw sentence "The cat sat" all the way to the final output vector Z, built step by step with animated Manim visuals and real matrix math. ━━━━━━━━━━━━━━━━━━━━━━ Timstamps: ━━━━━━━━━━━━━━━━━━━━━━ 0:06 Why Self-Attention 1:44 How Self-Attention Works (Mathematical Explanation) 9:13 Attention Heatmap 10:12 Full Self-Attention Pipeline 11:22 Outro ━━━━━━━━━━━━━━━━━━━━━━━ ✅ WHAT YOU WILL LEARN ━━━━━━━━━━━━━━━━━━━━━━━ ✅ Why sequential models (RNNs) fail at long-range dependencies and how self-attention solves this ✅ The full math behind Q, K, V projections, scaled dot-product attention (Q·Kᵀ / √dₖ), and softmax normalisation ✅ How to read an attention heatmap and understand what the model is actually "looking at" ━━━━━━━━━━━━━━━━━━━━━━━ 👤 WHO THIS IS FOR ━━━━━━━━━━━━━━━━━━━━━━━ This breakdown is for anyone who has heard of Transformers, ChatGPT, or large language models and wants to understand the actual mechanism — not just the metaphors. Prior knowledge of basic linear algebra (matrix multiplication) is helpful but not required. Every step is shown visually. ━━━━━━━━━━━━━━━━━━━━━━━ 📺 MORE FROM APPLIE AI LAB ━━━━━━━━━━━━━━━━━━━━━━━ Subscribe to Visual AI for weekly deep-dives into AI and machine learning concepts Next up: Multi-Head Attention explained the same way. #SelfAttention #AttentionMechanism #TransformerArchitecture #DeepLearning #NeuralNetworks #NaturalLanguageProcessing #MachineLearning #AIExplained #LargeLanguageModels #ManimAnimation
ADVERTISEMENT
But how are these Q, K, and V matrices determined?
This video solded a huge problem to me, I was struggling during months to understand how attention mecanism works, I always find myself lost between equations and weird terminology, finally found someone who explains in a simple direct way, Thank you a lot sir
I‘m interested to search more about the background of those Q, K, V, matrices. Why 3 why not 4, what was the idea behind… without searching it looks to me as the standard 3 layer data separation: source-manipulation-output. It is interesting that almost all data architectures uses 3 layers to be universal but not too complex.
For anyone curious, this video and code basic's transformer video are the two best explanations I have seen. They have truly helped me understand the concepts.
Can you explain cross attention, kv cache and the flow during inference?
nice walkthrough. but 6:19 - it felt so strange to include that. we know we're listening to AI, we don't need it to pretend it has a throat.
Well done video. I like AI visualizations and subscribed to the channel. Other new AI models could be interesting too. V-JEPA is key in the world model and is based on Transformer. Yann LeCun's new "Hierarchical Planning with Latent World Models" has been published. Maybe in the future there will be more than 2 hierarchical levels.
Excellent explanation. I am figuring this stuf out to improve my working with AI.
Excellent content. Please come up with more such topics
Wonderfully explained. I’ve never seen such a coherent and succinct explanation of all these topics, bravo. I think it’s quite interesting that words, like natural images, have unique statistical properties, and how those properties reveal themselves through seemingly simple ideas like correlation. Seems like LLMs aren’t truly learning context in this manner, but rather, they are learning statistical relationships between words. This is not quite context to me, but it is sufficient enough for LLMs to do well. A lot of similarities in the idea of efficient coding and the study of natural image statistics to understand how the visual system leverages properties of the world to form an efficient representation of the visual signal itself.
Amazing, can you sometime make video on diffusion model. Thanks
I don't know why people think self-attention is elegant. It's a clunky and computationally inefficient mechanism for (crudely) simulating semantic comprehension (although it does work). I'd be more impressed with a network which really understood that streets can't get tired (to use the sentence in this video).
Finally i get it!
7:55 I'm surprised you didn't also poibt out cat sat > sat cat
what's the need of value vectore are are we doing matmul with with value vector
lovely! i was expecting u to show the final attention matrix for the origonal sentence
Excellent! Please create a full course on this
Excellent video. Helped explain Self Attention really well. Thank you for creating this
Awesome video. Great examples and illustrations.
thank you. I finally understand