Training Trains logo

Inference Engineering Internship

Make AI models fast and affordable to run: latency and throughput, batching, quantisation, caching, serving frameworks and cost models.

PythonPyTorchHugging FaceONNX Runtimellama.cpp / OllamavLLM

What you'll work on

Full syllabus: all 60 graded tasks

These are the actual tasks the internship portal assigns for this domain. You submit each one with proof (a GitHub link or document) and a mentor reviews it.

Level 1: Foundations (24 tasks)
  1. What inference engineering is. Explain training vs inference, and why latency, throughput and cost matter when serving models.
  2. Python and PyTorch setup. Set up Python with PyTorch (CPU is fine) and run inference with a small pre-trained model.
  3. Hugging Face pipelines. Run text classification, summarisation and embeddings with Hugging Face pipelines and time each one.
  4. Tokens and tokenisers. Tokenise 5 sentences in English and an Indian language and explain why token counts differ.
  5. Measure latency. Measure first-token latency and total latency for a small language model over 20 prompts.
  6. Batching. Compare one-at-a-time vs batched inference for an image or text model and report throughput.
  7. Serve with FastAPI. Wrap a model in a FastAPI endpoint and call it from a script.
  8. Streaming responses. Stream generated tokens to the client with server-sent events.
  9. Hardware basics. Explain CPU vs GPU vs specialised accelerators for inference, and what memory bandwidth means.
  10. Model size and memory. Calculate how much memory a 7B-parameter model needs in fp32, fp16 and int4.
  11. Use an LLM API. Call a hosted LLM API, measure latency and cost per 1,000 requests for a fixed prompt.
  12. Repo and benchmarks. Keep all scripts and benchmark results in a repo named inference-internship with a results table.
  13. Model zoo comparison. Compare 5 small open models on speed, size and answer quality for one task.
  14. Precision experiment. Run the same model in fp32 and fp16 (or bf16) and compare speed and output.
  15. Tokeniser cost table. Build a table of token counts and API cost for 10 prompts in English and Indian languages.
  16. Timeouts and retries. Add timeouts and retries with backoff to your inference client.
  17. Request validation. Validate prompt length and parameters before calling the model.
  18. Benchmark script. Write a reusable benchmark script that reports P50, P95 and throughput.
  19. CPU inference tuning. Tune thread counts for CPU inference and record the best setting.
  20. Warm-up. Add model warm-up at startup and measure first-request latency before and after.
  21. Health and readiness. Add readiness checks that pass only after the model is loaded.
  22. Simple queue. Put a queue in front of the model so bursts do not crash it.
  23. Logging tokens. Log input and output tokens per request and total per day.
  24. Cost dashboard. Chart daily cost from your token logs.
Level 2: Building real features (20 tasks)
  1. Quantisation. Quantise a model (bitsandbytes, GGUF or ONNX) and compare speed, memory and output quality.
  2. ONNX Runtime. Export a model to ONNX and compare its speed with PyTorch.
  3. Local LLM serving. Run a small open LLM with llama.cpp or Ollama and benchmark tokens per second.
  4. KV cache. Explain the key-value cache and show how output length affects memory and speed.
  5. Load testing. Load-test your endpoint with k6 or Locust and find the point where latency becomes too high.
  6. Response caching. Cache repeated prompts and measure the hit rate and savings.
  7. Prompt and context length. Measure how latency and cost grow with prompt length and suggest limits.
  8. Embeddings service. Build a fast embeddings endpoint with batching and use it for a small semantic search.
  9. Containerised serving. Put the model server in Docker with health checks and resource limits.
  10. Observability. Log latency, tokens, errors and cost per request and chart them.
  11. Prompt caching. Use provider prompt caching or a prefix cache and measure savings.
  12. Semantic caching. Cache answers for similar questions using embeddings and measure accuracy risk.
  13. Model routing. Route easy requests to a small model and hard ones to a large model and compare cost and quality.
  14. Structured output. Force JSON output with a schema and measure failure rates.
  15. Batch API usage. Use a provider's batch API for offline jobs and compare cost with live calls.
  16. GPU memory profiling. Measure GPU memory for different batch sizes and context lengths.
  17. Throughput vs latency curve. Plot throughput against latency for different concurrency levels.
  18. Fallbacks. Add fallback to a second model or provider when the first fails.
  19. Edge inference. Run a small model on a phone or Raspberry Pi and benchmark it.
  20. Whisper / speech inference. Benchmark a speech-to-text model and optimise its speed.
Level 3: Professional work and capstone (16 tasks)
  1. vLLM or TGI. Serve a model with vLLM or Text Generation Inference (or a cloud GPU notebook) and compare with your basic server.
  2. Continuous batching. Explain continuous batching and paged attention and show their effect in a benchmark.
  3. Speculative decoding concept. Explain speculative decoding and estimate its speed-up for a draft/target model pair.
  4. Autoscaling design. Design autoscaling for a model service based on queue length and GPU use.
  5. Cost model. Build a spreadsheet comparing self-hosting vs API pricing at 1k, 100k and 1M requests a day.
  6. Quality vs speed evaluation. Build a small evaluation set and compare answer quality across your faster model variants.
  7. Safety and limits. Add input limits, rate limits and output filtering to your serving endpoint.
  8. Capstone: production inference service. Deliver an optimised inference service with benchmarks, quantisation results, monitoring, a cost model and a design document.
  9. Tensor parallelism concept. Explain tensor and pipeline parallelism and when a model needs multiple GPUs.
  10. TensorRT or OpenVINO. Optimise a model with TensorRT, OpenVINO or a similar toolkit and compare speed.
  11. Fine-tuned small model. Fine-tune a small model with LoRA for one task and compare its cost with a large general model.
  12. SLA design. Define latency and availability SLAs for an AI feature and design to meet them.
  13. Multi-tenant serving. Serve several customers from one model service with fair rate limits.
  14. Load shedding. Reject or degrade requests gracefully when the system is overloaded.
  15. Inference postmortem. Write a postmortem for a simulated latency spike.
  16. Capstone 2: cost-optimised AI feature. Ship an AI feature that meets a latency target at the lowest cost, with routing, caching, monitoring and a cost report.

Real software work, not just training

Every Inference Engineering task is reviewed by our team and saved to your portfolio. Interns who pass the readiness check can take real work from Training Trains, W3AppDevelopers and DomainHostly projects on the Company Work board.

AI / Machine Learning Solutions
See the software we build →

Frequently asked questions

Do I need experience to join the Inference Engineering internship?+

Python and basic machine learning. A GPU helps but free cloud notebooks are enough.

What will I do in the Inference Engineering internship?+

You complete 60 graded tasks in three levels: foundations, real features, and professional work. Each task is submitted with GitHub or document proof and reviewed. The final capstone is: Capstone 2: cost-optimised AI feature.

Do I get a certificate for the Inference Engineering internship?+

Yes. After your required tasks are approved, your attendance is above 75% and you pass the proctored final exam, you get a graded completion certificate that anyone can verify at trainingtrains.com/verify.

Can I do the Inference Engineering internship online?+

Yes. You can do it fully online from home, or offline at our Erode and Perundurai branches. Tasks, quizzes and mentor reviews are the same in both modes.

Available both online (work from home) and offline at our Erode and Perundurai branches - see the full internship program details for departments, duration, and the application process.

Apply for the Inference Engineering internship