MLE-430: Model Serving and Inference Infrastructure

Course Description

Production inference for ML models. ONNX and model export. NVIDIA Triton Inference Server: model repository, batching, concurrency. vLLM for LLM serving: continuous batching, paged attention, quantization support. TGI (Text Generation Inference) by HuggingFace. BentoML for lightweight serving. Autoscaling: Kubernetes HPA, KEDA, GPU autoscaling. Latency optimization: caching, request batching, model distillation. Cost modeling and ROI calculations for inference infrastructure.