Meta dropped Llama 3.3, a 405-billion-parameter behemoth distilled into a lean, mean, 70-billion-parameter machine. Reminds me of the "GPT-4 to GPT-4o to GPT-4o mini" saga. Same playbook, new players.

Seems like the formula is clear:

1. Build an outrageously massive model.

2. Figure out the most important parts of the model by observing core usage patterns.

3. Distill it down until it is cheaper, faster, and far more usable.

Distillation is not just the trend; it is going to be a cult given how expensive naked Transformers are.

DeepSeek's Distilled Models: The Real Breakthrough

My DeepSeek highlight was not the headline model. It was the performance of the distilled smaller models. Look at DeepSeek-R1-Distill-Llama-70B kicking o1-mini's performance on benchmarks. Like for like, o1-mini costs 12 times Llama-70B. Let that sink in.

Why We Have Hit the Ceiling of AI Scaling

Ilya Sutskever, ex co-founder of OpenAI, recently shared his thoughts. Three key takeaways:

1. We Have Reached Peak Data

The internet, the entirety of digitized human knowledge, is finite. We have reached peak data, and the returns from scaling alone have hit a limit.

2. The Path Forward Is Uncertain

While Ilya outlined three areas of focus (agents, synthetic data, and optimizing inference), his tone felt uncertain. His tone mimics what the larger deep learning community is grappling with: translating expensive transformer-based systems to real business value.

3. Better Reasoning Means More Hallucinations

Future systems will reason better, and with great reasoning comes greater unpredictability (read: hallucinations). This unpredictability is already causing unease among businesses, and it will be fascinating to see how adoption cycles evolve.

The AI Pricing Problem Nobody Can Solve

$200 a month for what feels like marginally better reasoning? Charging that for "delta better reasoning" seems like trying to sell Ferraris to farmers for plowing fields. Yes, it is powerful, but who is actually going to pay for that?

And then the enterprise angle: $60 per million output tokens for the API? That is steep. Pay with an arm and a leg for a reply that may or may not be correct.

This pricing reality is exactly why control, not the model, is the real differentiator. And it is why Meta's strategic acquisition of Manus focused on distribution and infrastructure, not on building the smartest model.

Cheers to smaller, smarter, more effective AI. The era of bloated models is so 2023.

Frequently Asked Questions

What is AI model distillation?

Distillation is the process of training a smaller, faster model to replicate the performance of a larger model. The large model's knowledge is compressed into a smaller architecture that is cheaper to run and often nearly as capable for practical use cases.

Is DeepSeek better than OpenAI for most use cases?

DeepSeek's distilled models offer comparable performance to OpenAI's models at a fraction of the cost. For many practical business applications, the cost-performance ratio of distilled open-source models is now superior to premium closed-source alternatives.

Why did Ilya Sutskever say AI scaling has hit a limit?

The core data available for training (the internet) is finite. Beyond a certain scale, adding more parameters and data yields diminishing returns. The industry is shifting focus from bigger models to smarter training methods, better inference, and domain-specific fine-tuning.