How to Deliver a 128-GPU Cluster (K8s + Slurm Hybrid · 7 Steps)¶

TL;DR: A 128-GPU cluster is not "just buy A800". It's rack + cooling + orchestration + observability + storage + preemption + rollout — seven independent disciplines. Skip any one and you'll waste months in production stabilization.

Real numbers from a delivered AI research institute project (12 teams, 128 GPUs = A800 + 4090): - GPU utilization 32% → 71% - Training queue time -60% - 12 teams shifted from complaints to voluntary adoption

See Case 13 for the full case.

Step 1 · Rack + Power + Cooling¶

128 A800 ≈ 40-80 kW/rack. Standard data-center racks (10-15 kW) will overheat. Plan in-row cooling or liquid loop at rack level.

• Cabinet spec: rack ≥ 40 kW, ideally 60 kW
• Cooling: chilled water (in-row) or immersion (for A100 / H100 dense)
• PDU: dual-path, 32A / 63A per socket
• Do not try to fit A800 into a legacy server room without HVAC upgrade

Step 2 · NVIDIA GPU Operator + Slurm alongside K8s¶

Two workloads, one GPU pool at driver layer:

• K8s for online inference / short tasks / elastic scaling
• Slurm 22+ for offline training / long tasks / preemptible

Both talk to NVIDIA driver via GPU Operator — no double-installation of CUDA / drivers per node.

Alternative: pure K8s with training operator (Kubeflow) — works but training UX is degraded compared to Slurm for large teams.

Step 3 · Observability First (Wire Before Traffic)¶

• DCGM Exporter (per-GPU metrics: utilization, VRAM, temp, power)
• Prometheus + Grafana (retention 30d hot / 1y cold)
• LLM API cost dashboard (per team / per model / per token type)
• Alert rules (utilization < 30% for 24h = tuning opportunity, temp > 85°C = incident)

Wire ALL of the above before production traffic. If observability arrives after prod, you're debugging blind in prod outages.

Step 4 · Storage Tiering¶

Storage IO is a frequent bottleneck, more than GPU.

• Local NVMe on each node: model weights + LORA (first-load must not hit network)
• NFS hot: recent 3 checkpoints (fast rollback)
• Ceph cold: archive checkpoints (7d+)
• Don't put all checkpoints on one NFS — a training job dumping 300GB will freeze everyone

Step 5 · Preemption Strategy¶

K8s online inference wins over Slurm training on peak.

• Priority classes in K8s (production > staging > dev)
• Slurm QoS: training preempts training, never inference
• Gang scheduling for multi-GPU training (don't hold GPUs waiting for peers)

Without preemption strategy: 30% wasted GPU at peak while training crowds out inference. Users escalate.

Step 6 · 168h Stability + 3× Peak Shock Load-Test¶

Not a one-off acceptance test on day of go-live.

• 168 hours continuous load (7 days simulated production)
• 3× peak concurrent shock (training + inference + storage IO all peaking together)
• Every failure mode logged: single-node fail, network partition, storage jitter, model hot-swap
• Every failure has a documented SOP before rollout

Step 7 · Gradual Rollout¶

• Week 1-2: canary team (1-2 teams, low-risk workloads)
• Week 3-6: half teams (6 teams, mixed workloads)
• Week 7+: full rollout (12 teams, all workloads)

Do not switch all 12 teams overnight. First-week teething = if 12 teams hit at once, escalations flood the SRE inbox and rollback is politically hard.

Common Failure Modes¶

• ❌ Bought A800 without upgrading rack cooling → derating, real utilization stays at 30%
• ❌ Pure K8s for training → training UX bad, researchers rebel
• ❌ No storage tiering → first-query slow, users abandon
• ❌ No preemption strategy → inference stalls during training peaks
• ❌ Rollout to all 12 teams day-1 → first-week issues escalate to CEO

Related¶

• Case 13 · 128-GPU K8s+Slurm
• Direction · AI + DevOps
• Private AI sizing (6 steps)
• Portfolio PDF (200+ AI deliveries)

Next: 30-min cluster design alignment · Download portfolio PDF