Private AI deployment sizing rules

Do not start a private AI deployment project by picking a GPU. Start by answering these questions in order. If you skip the earlier ones, hardware selection is guessing.

1. Peak concurrency and p95 latency

• What is the expected peak concurrent user count?
• What is the acceptable p95 response latency (first token and full response)?
• What is the request pattern — burst, sustained, or long tail?

These three numbers set the minimum feasible model size and batching strategy before any hardware discussion.

2. Data boundary

• Does the input data have to stay on-premise, in-region, or in a specific tenant?
• Which regulator's rules apply (finance, healthcare, government, industrial, cross-border)?
• Which parts of the pipeline are allowed to call out (embedding, moderation, TTS)?

The boundary determines whether you can use a hosted gateway at all. Some deployments only need private inference while allowing hosted embedding; others require full air-gap.

3. Cost ceiling and failure ownership

• Monthly cost cap for hardware + power + ops.
• What happens if the model goes down at 3 AM — who is paged and with what SLA?
• What happens if the model regresses after a driver / weight update — how do we roll back?

If ownership is not written down, private deployment will drift from "cheaper than API" back to "silently more expensive than API" within one quarter.

4. Monitoring, degradation, and cost alarms

Required at minimum:

• Per-request latency histogram, per-endpoint.
• Token throughput and GPU utilization.
• Automatic degrade path (smaller model, or fallback to hosted API) when p95 breaches SLA.
• Cost alarm before, not after, the monthly ceiling is hit.

5. Agent permissions minimization

Any AI agent that can write to production systems (databases, tickets, external APIs) must:

• Run under a dedicated, least-privilege identity.
• Have all destructive actions require a human confirmation or a signed policy.
• Log every tool call with input and output for audit.

6. Hardware — only now

• Choose model size based on 1–5, not brochure benchmarks.
• Pilot with vLLM or a comparable serving stack before signing a hardware PO.
• Stress test at 2x peak concurrency for 24 hours before signoff.
• Prefer hardware you can extend or replace, not one-off silicon.

Anti-patterns

• "Buy 8 x H100 first, then we figure it out."
• "The vendor demoed a chat window, therefore it will handle production."
• "We do not need monitoring, the model is fine."
• "Fine-tuning will fix the accuracy problem" without a labeled evaluation set.

Private deployment is a total cost of ownership question, not a GPU question.