How to Size a Private AI Deployment (6 Steps · Before You Buy Any GPU)¶
TL;DR: Skipping any of these 6 steps = 30-60% wasted GPU or 2-3 wasted delivery months. Applies to healthcare / manufacturing / law firm / SOE / finance / cross-border SaaS.
Step 1 · Quantify Peak Concurrency¶
Peak QPS + parallel long-context sessions + duration distribution. Not average — peak. If unknown, worst case = 2× your best guess.
Sample real numbers from delivered projects: - Tier-3 hospital doctor assistant: ~30 concurrent doctors at peak (morning rounds 8-10am), each session 200-2000 tokens output. See Case 01. - 400+ microservice AIOps: ~5000 alerts/day, peak concentrated in incidents. See Case 14. - 200+ engineer CI/CD: ~50k commits/month, peak on Monday and pre-release. See Case 15.
Step 2 · Freeze p95 Latency Target¶
Pick before hardware. User pain threshold: - Chat / doctor query: 2-3s p95 - Voice interaction: 800ms p95 - Line-side vision (visual QA): 100ms p95 - Bulk analysis (contract review): 60s batch is acceptable
Latency target directly drives model size cap. 72B at 3s p95 requires ≥ 2×A800 or ≥ 4×4090.
Step 3 · Profile Task Shape¶
Task shapes have very different infra needs:
| Task shape | VRAM | Throughput priority | Context |
|---|---|---|---|
| Short chat (Q&A) | Low | Concurrency | ≤ 8k |
| Long RAG (multi-document) | Medium | Balanced | 8-32k |
| Long context (tenders / contracts) | High | Latency | 128k+ |
| Training / fine-tune | Very high | Throughput | N/A |
Do not mix training and inference on the same GPUs without K8s + Slurm hybrid orchestration. See Case 13 for a 128-GPU real setup.
Step 4 · Pick Model + Embedding + Reranker Triple¶
All three together. Model in isolation is meaningless.
| Task | Model | Embedding | Reranker |
|---|---|---|---|
| Doctor guideline lookup | AW36-72B | bge-large | BGE-reranker |
| Air-gapped factory RAG | AW36-14B | bge-large | (single-tenant, skip) |
| Cross-border SaaS (cost-optimized) | AW36-Max + Claude 3.5 | m3e | BGE-reranker |
Step 5 · Size Storage IO¶
Storage IO is often the bottleneck, not GPU.
- Vector index: current size × 3-5 for growth. Milvus HNSW default eats 3× raw vector size.
- Checkpoint tiered storage: NFS hot (recent 3 checkpoints) + Ceph cold. Do not put all checkpoints on one NFS.
- Model weights: keep on local NVMe, not network — first-load takes minutes if network.
Step 6 · Set Cost Cap Upfront¶
If nothing is capped, prod cost balloons 3-10× in 6 months.
- Per-department token quota (Redis metering)
- Monthly budget circuit-breaker (auto-suspend at 80% of budget)
- Per-request soft cap (kick to smaller model beyond N tokens)
- Audit log 100% coverage (regulator can query any time)
See Case 03 for a group-wide multi-tenant gateway with all three enforced.
Only Then, Decide Hardware¶
| Signal | Hardware |
|---|---|
| 72B + high concurrency | 4-8×A800 (vLLM) |
| 14B-32B + single-tenant | 1×4090 / RTX 6000 Ada |
| Training + inference share | K8s + Slurm hybrid on A800 cluster |
| Air-gapped workshop | Single 4090 edge |
| Cost-optimized cross-border | Public API + selective on-prem gateway |
Common Failure Modes¶
- ❌ Buy hardware first, then figure out concurrency → 30-60% wasted GPU
- ❌ Pick a model without embedding/reranker → RAG accuracy 55-70% (unacceptable for production)
- ❌ Ignore storage IO → first-query slow, users abandon
- ❌ No cost cap → runaway budget, department screams at CIO
- ❌ Mix training + inference on same GPUs without hybrid orchestration → prod inference stalls during training peaks
Related¶
- Private AI deployment sizing (canonical page)
- 20 Case Studies with hard KPIs · PDF
- Vendor acceptance checklist