Direction 5 · AI + DevOps (AIOps · MLOps · CI/CD)¶
Productionizing AI systems = observability + load-test + fallback + circuit-breaker + rollback. Not "if it runs, ship it". 3 representative cases + delivery discipline.
Core Capabilities¶
- GPU clusters: K8s + Slurm hybrid · NVIDIA GPU Operator · MIG · DCGM
- Observability: Prometheus · Grafana · OpenTelemetry · ClickHouse · Loki
- CI/CD: GitLab / GitHub Actions · LLM code review · unit-test generation
- AIOps: logs + metrics + traces triple-source fusion · LLM root-cause correlation
- Storage: NFS · Ceph · Lustre · GPU passthrough · checkpoint tiering
3 Representative Cases¶
Case 01 · GPU Cluster K8s + Slurm 128-GPU Training Platform¶
- Problem: 12 AI teams sharing cluster · GPU util only 32% · training queue often 3-7 days
- Stack: K8s (online inference) + Slurm 22 (offline training) hybrid · NVIDIA GPU Operator · DCGM
- Key engineering: preemption · colocation isolation · GPU passthrough · VRAM fragmentation · checkpoint tiering · priority queue
- KPI: GPU utilization 32% → 71% · training queue time -60% · 12 teams onboarded
- My role: platform architecture · K8s+Slurm fusion · hardware selection · load test · SLA
Case 02 · AIOps Root-Cause Analysis (Logs + Metrics + Traces Fusion)¶
- Problem: 400+ microservices · 5000+ alerts/day · root-cause depended on senior SREs manually correlating
- Stack: OpenTelemetry ingest · ClickHouse storage · LLM root-cause correlation · SRE feedback loop
- Algorithm: triple-source time alignment · LLM hypothesis generation · Top-3 candidate ranking · SRE feedback retraining
- KPI: MTTR 45min → 12min · alert fatigue halved · SRE team pivots from firefighting to reliability engineering
- My role: data-flow architecture · LLM correlation engineering · SRE integration
Case 03 · CI/CD AI Code Review + Unit-Test Auto-Generation¶
- Problem: 200+ engineers · 50k commits/month · Code Review P50 24 hours · high-risk changes leaking through
- Stack: GitLab webhook · LLM router (cost-sensitive → DeepSeek · complex logic → GPT-4o) · unit-test generation
- Strategy: static analysis + recent-commit correlation + rule library match · high-risk to human · mid/low auto-approve
- KPI: Code Review P50 24h → 1h · defect escape double-digit reduction · engineer satisfaction +30%
- My role: platform architecture · CI/CD plugin · model routing · effect acceptance
Delivery Discipline¶
- Hardware selection front-loaded: 128 GPUs = peak concurrency · p95 · training shape · storage IO all quantified BEFORE buying
- Observability triple-layer: DCGM (GPU) + Prometheus (system) + LLM API cost tracking
- Load test: 168h stability + peak 3× shock — no one-shot acceptance
- Failure fallback: single-node · network partition · storage IO jitter · model hot-swap — every failure has a documented SOP
- Cost caps: department quota + per-task cap + daily budget circuit-breaker — 3-layer combined
Common Anti-Patterns¶
- ❌ K8s + GPUs but no GPU Operator — GPUs can't be scheduled
- ❌ Training checkpoints all on local NFS — IO saturated
- ❌ Prometheus monitors CPU/RAM only — GPU temp / VRAM ignored → hardware degrades
- ❌ CI/CD uses GPT-4 with no quota — one review costs $5
Related¶
- Vendor acceptance checklist (8 hard KPIs)
- Full 20-case portfolio · PDF
- Related directions: Private LLM deployment · Multi-model routing