Training Trains logo

Inference Engineering Terms Every Intern Should Know

By K. Boopathikumar, CEO & Founder · 16 November 2026 · 4 min read

In your first Inference Engineering internship meetings you will hear words that everyone else seems to know. Here they are in plain English. If you hear a term that is not on this list, write it down and ask; nobody minds an intern asking what a word means.

The terms

  • Latency: time for one request.
  • Throughput: requests handled per second.
  • p99: the latency 99% of requests are faster than.
  • Batching: processing several requests together.
  • Quantisation: lower-precision weights for speed.
  • ONNX: a portable model format.
  • Cold start: delay when a new server or model loads.
  • Cost per request: hardware cost divided by requests served.

Next steps

Looking for an internship near Erode?

10 days to 6 months, online or at our Erode and Perundurai branches, with a verifiable certificate.

See internships →

More guides