Module 6 · Wrap-up¶
What you built¶
Three and a half hours ago your cluster had never seen a GPU. Now:
- A Karpenter NodePool that buys g5/g6 capacity on demand, Spot first, on a pinned AMI, with a hard GPU limit and a disruption policy chosen on purpose
- The NVIDIA GPU Operator, configured the way EKS needs it (
driver.enabled=false,toolkit.enabled=false) — and you know why - A node you time-sliced four ways, saw four processes share one card, and un-sliced again
- vLLM serving a real model through an OpenAI-compatible endpoint with probes that don't kill it
- KEDA scaling on requests in flight, and Karpenter adding a GPU node behind it while you timed every phase of the cold start
- DCGM → Prometheus → Grafana, and a one-line query that finds idle GPUs
- Two idle nodes reclaimed by changing one field
Everything reruns from the companion repo: make workshop-up builds all of it; make workshop-down tears it down.
What we deliberately didn't cover¶
| Topic | One line | Start here |
|---|---|---|
| Distributed training | EFA networking, NCCL, p4/p5 capacity blocks, Kubeflow/Ray on EKS. A different workshop. |
EKS best practices: AI/ML |
| Dynamic Resource Allocation | The device-plugin successor: GPUs as first-class scheduler objects. GA in 1.34; AWS recommends it for static capacity today. | Kubernetes DRA · EKS NVIDIA DRA driver |
| MIG in practice | Hardware partitioning on A100/H100. Same Operator, nvidia.com/mig.config label, mig-1g.10gb resources. |
GPU Operator: MIG · EKS: MIG |
| Multi-cluster GPU fleets | Capacity in three regions, one control plane for scheduling. | Karmada / Kueue MultiKueue |
| Inference gateways | Model-aware routing, prefix-cache affinity, KV-aware load balancing — the layer above a Service. | Kubernetes Gateway API Inference Extension · llm-d |
| EKS Auto Mode for GPUs | Everything from Lab 1, managed by AWS. Fewer knobs; the ones you keep are NodePool/NodeClass. | EKS Auto Mode: accelerated instances |
Take-aways¶
- Production readiness checklist — every decision from today as a yes/no you can run against your own cluster.
- Where to go next — the official docs everything in this guide was verified against.
Before you go¶
- Clean up. Do it now, together — it takes five minutes and the GPU nodes are the expensive part.
- Feedback — the link is in the chat. Two minutes. What should the next version of this workshop do differently?
- Stay in touch — abebars.io ·
#LevelUpWithAhmed
Q&A¶
Ask anything. The answers that need a terminal are welcome; the cluster is still up until step 1.