All 60 tasks, level by level

The real graded Inference Engineering tasks the portal assigns, grouped into three levels.

The portal assigns these Inference Engineering tasks. Finish Level 1 before Level 2 and Level 2 before Level 3. Your mentor may mark some tasks as required for the certificate.

Level 1: Foundations (24 tasks)

Task	What to do
What inference engineering is	Explain training vs inference, and why latency, throughput and cost matter when serving models.
Python and PyTorch setup	Set up Python with PyTorch (CPU is fine) and run inference with a small pre-trained model.
Hugging Face pipelines	Run text classification, summarisation and embeddings with Hugging Face pipelines and time each one.
Tokens and tokenisers	Tokenise 5 sentences in English and an Indian language and explain why token counts differ.
Measure latency	Measure first-token latency and total latency for a small language model over 20 prompts.
Batching	Compare one-at-a-time vs batched inference for an image or text model and report throughput.
Serve with FastAPI	Wrap a model in a FastAPI endpoint and call it from a script.
Streaming responses	Stream generated tokens to the client with server-sent events.
Hardware basics	Explain CPU vs GPU vs specialised accelerators for inference, and what memory bandwidth means.
Model size and memory	Calculate how much memory a 7B-parameter model needs in fp32, fp16 and int4.
Use an LLM API	Call a hosted LLM API, measure latency and cost per 1,000 requests for a fixed prompt.
Repo and benchmarks	Keep all scripts and benchmark results in a repo named inference-internship with a results table.
Model zoo comparison	Compare 5 small open models on speed, size and answer quality for one task.
Precision experiment	Run the same model in fp32 and fp16 (or bf16) and compare speed and output.
Tokeniser cost table	Build a table of token counts and API cost for 10 prompts in English and Indian languages.
Timeouts and retries	Add timeouts and retries with backoff to your inference client.
Request validation	Validate prompt length and parameters before calling the model.
Benchmark script	Write a reusable benchmark script that reports P50, P95 and throughput.
CPU inference tuning	Tune thread counts for CPU inference and record the best setting.
Warm-up	Add model warm-up at startup and measure first-request latency before and after.
Health and readiness	Add readiness checks that pass only after the model is loaded.
Simple queue	Put a queue in front of the model so bursts do not crash it.
Logging tokens	Log input and output tokens per request and total per day.
Cost dashboard	Chart daily cost from your token logs.

Level 2: Building real features (20 tasks)

Task	What to do
Quantisation	Quantise a model (bitsandbytes, GGUF or ONNX) and compare speed, memory and output quality.
ONNX Runtime	Export a model to ONNX and compare its speed with PyTorch.
Local LLM serving	Run a small open LLM with llama.cpp or Ollama and benchmark tokens per second.
KV cache	Explain the key-value cache and show how output length affects memory and speed.
Load testing	Load-test your endpoint with k6 or Locust and find the point where latency becomes too high.
Response caching	Cache repeated prompts and measure the hit rate and savings.
Prompt and context length	Measure how latency and cost grow with prompt length and suggest limits.
Embeddings service	Build a fast embeddings endpoint with batching and use it for a small semantic search.
Containerised serving	Put the model server in Docker with health checks and resource limits.
Observability	Log latency, tokens, errors and cost per request and chart them.
Prompt caching	Use provider prompt caching or a prefix cache and measure savings.
Semantic caching	Cache answers for similar questions using embeddings and measure accuracy risk.
Model routing	Route easy requests to a small model and hard ones to a large model and compare cost and quality.
Structured output	Force JSON output with a schema and measure failure rates.
Batch API usage	Use a provider's batch API for offline jobs and compare cost with live calls.
GPU memory profiling	Measure GPU memory for different batch sizes and context lengths.
Throughput vs latency curve	Plot throughput against latency for different concurrency levels.
Fallbacks	Add fallback to a second model or provider when the first fails.
Edge inference	Run a small model on a phone or Raspberry Pi and benchmark it.
`Whisper / speech inference`	Benchmark a speech-to-text model and optimise its speed.

Level 3: Professional work and capstone (16 tasks)

Task	What to do
`vLLM or TGI`	Serve a model with vLLM or Text Generation Inference (or a cloud GPU notebook) and compare with your basic server.
Continuous batching	Explain continuous batching and paged attention and show their effect in a benchmark.
Speculative decoding concept	Explain speculative decoding and estimate its speed-up for a draft/target model pair.
Autoscaling design	Design autoscaling for a model service based on queue length and GPU use.
Cost model	Build a spreadsheet comparing self-hosting vs API pricing at 1k, 100k and 1M requests a day.
Quality vs speed evaluation	Build a small evaluation set and compare answer quality across your faster model variants.
Safety and limits	Add input limits, rate limits and output filtering to your serving endpoint.
Capstone: production inference service	Deliver an optimised inference service with benchmarks, quantisation results, monitoring, a cost model and a design document.
Tensor parallelism concept	Explain tensor and pipeline parallelism and when a model needs multiple GPUs.
TensorRT or OpenVINO	Optimise a model with TensorRT, OpenVINO or a similar toolkit and compare speed.
Fine-tuned small model	Fine-tune a small model with LoRA for one task and compare its cost with a large general model.
SLA design	Define latency and availability SLAs for an AI feature and design to meet them.
Multi-tenant serving	Serve several customers from one model service with fair rate limits.
Load shedding	Reject or degrade requests gracefully when the system is overloaded.
Inference postmortem	Write a postmortem for a simulated latency spike.
Capstone 2: cost-optimised AI feature	Ship an AI feature that meets a latency target at the lowest cost, with routing, caching, monitoring and a cost report.

← Tools and first-week setup Submitting tasks and earning your certificate →