InternshipKey terms and common mistakes
The words you will hear in Inference Engineering work and the mistakes that most often cost interns approvals.
Key terms
| Term | Meaning |
|---|---|
| Inference | Running a trained model to get predictions or text |
| Latency | Time from request to response |
| Throughput | Requests or tokens served per second |
| Batching | Processing several requests together for efficiency |
| Quantisation | Using lower-precision numbers so a model runs faster and smaller |
| KV cache | Stored attention data that speeds up text generation |
| GPU utilisation | How busy the GPU is |
| Model serving | Running a model behind an API for many users |
Common mistakes to avoid
- Measuring speed on one request instead of real load.
- Ignoring memory limits until the server crashes.
- Quantising without checking quality loss.
- No monitoring of latency and errors in production.
Want to go deeper? The free Deep Learning learning path has short modules, games and a certificate for each module.