LLM Compression Explained: Build Faster, Efficient AI Models

Here I would stress that everything is a tradeoff. And while it was said in the video that you are loosing accuracy by the quantization, I still feel that it has to be repeated over and over that by doing this you are loosing accuracy and there might be use cases where accuracy on broad categories of topics is very important. Also in most cases you need a AI model that is a specialist on just few topics / things. So reducing the number of model parameters would be also a path to think about. But in regards to the optimization - how much to scale down the size of each parameter would be a nice topic to touch upon.

Thanks for watching folks! Apologies about the small verbal typo there, meant to say 10 GPU's :)

Great video, thanks, looking for more

Easy to understand. Thank you

My life is better thanks to IBM Technology. Thank you <3

This is one of the few explanations that treats LLMs as real deployment systems where inference cost and latency dominate, especially in multi-agent architectures where delays compound across nodes. I’ve been building similar AI systems, and the quantization trade-offs here match exactly what makes the difference between a prototype and a production-ready system

Informational video thanks ! 😊

I would love to see a video about Vulkan, particularly with respect to its ability to run on a wide variety of different Hardware

Nice positive info thx

Very impressive

thanks

Perfect!

يجب أن تذكر ايضا الجانب السلبي أن الدقة تصبح أقل و ربما اخطاء تظهر ، شكرا على المعلومات القيمة ، رغم لا علاقة لي بالبرمجة الا انني استفيد لاني مهتم بالتقنية بشكل عام

How about TinyML. Any related to Model compression and quantization?

I ran qwen3 on my laptop without gpu with only 8gb ram and it ran smoothly. Of course my os is Linux.

Nice, so I can get 2+2=5 faster , instead of 2+2=4, juhu, User will be happy, right? Imagine a calculator with option to run faster but with no guarantee calculations are correct !

Uh, 800GB divided by 80GB/GPU equals ten GPUs by my arithmetic.... And that's just the weights. You'd need more for your KV cache, I think. So something like at least 12 80GB GPUs.

800 GB = 10 A100 (minimum) not five. That is also a $12,000 - $15,000 per month pile of GPUs , before you do anything with it.

Writing partial words & abbreviations on the whiteboard isn’t as helpful as the whole word. I recommend to either write the words on the board or don’t bother writing at all.

Great explanation! God bless everyone!