Knowledge Distillation Part 1

What is knowledge distillation?

Knowledge distillation is a technique used to transfer information from a large, complex model to a smaller, simpler model. The goal is to create a more efficient model that retains much of the performance of the larger model while being faster and less resource-intensive.

Why do we need transfer knowledge from a large complex model to a smaller simpler model?

1, Smaller models require less computational power and memory, enabling faster inference and lower energy consumption.

2, Simpler models can run on resource-constrained devices like mobile device or laptop.

3, So less computational requirements translate to lower cloud computing and storage costs for large-scale deployments.

4. Also lower latency compare to the call the server models.

The senario when the model distillation is used?

For example, in the recently published Apple Intelligence, Apple developed two foundation models: a larger server-based model and a smaller ~3 billion parameter model designed for on-device use.

Recognizing the challenges of running large models on mobile devices, Apple employed several techniques to optimize the on-device model, including knowledge distillation, model pruning, and quantization.

These approaches allowed Apple to create a highly capable on-device model that maintains strong performance across various benchmarks, while being compact enough to run efficiently on iPhones, iPads, and Macs.

Is this important for the interview?

Yes, it is important for the interview. The current AI trend is deeply updating the current tech infrastructure. Most companies need to upgrade their current infrastructure systems to incorporate AI technologies into each component.

For example, we will see each component adding more and more models. Model training and model inference will be everywhere in the system.

Serving these models at a large scale can be challenging. Adopting model distillation can be a good solution to address these challenges. It allows companies to create smaller, more efficient models that can run on various devices while maintaining strong performance.

(This approach is particularly valuable for deploying AI capabilities on mobile devices or in resource-constrained environments.)

The Knowledge Distillation Process Simplified

• The teacher model processes the training data.
• It produces outputs, including both hard predictions and soft probabilities.
• The student model is trained to mimic these outputs.

Interview Question using transfer knowledge:

Question1: Mobile Speech Recognition at Google

Google implemented knowledge distillation to improve its on-device speech recognition system for mobile devices.

Question Background:

Google's server-based speech recognition models were large and complex, providing high accuracy but requiring significant computational resources. These models couldn't run efficiently on smartphones due to their size and processing requirements.

Goal:

The goal was to create a speech recognition system that could run directly on mobile devices, providing fast, offline functionality while maintaining high accuracy.

High-level approach to the infra:

1. Started with their best server-based speech recognition models.
2. Use these large models to transcribe a massive amount of anonymized voice search queries.
3. The transcriptions from these powerful models, along with their confidence scores, were used as training data for smaller, mobile-optimized models.
4. The smaller models learned to mimic the behavior of the larger models, including their uncertainty on different inputs.

Results:

• The resulting mobile models were about 100 times smaller than the original server models.
• These models could run in real-time on mobile devices, even without an internet connection.
• Despite their small size, they achieved accuracy close to that of the larger server-based models.

Next Post

We will cover more about knowledge distillation process an more related interview questions. Stay tuned!

ML System Design Private Course 📒

• Do you need one-on-one training for ML System Design Interviews?
• Would you like to master modern AI infrastructure from FAANG engineers with 15+ years of experience? Contact us at [email protected]