Direction 1 · Private LLM Deployment (incl. AW36)¶
Private LLM is a total-cost + data-boundary + failure-accountability problem — not a GPU shopping list. 3 representative cases + delivery discipline.
Core Capabilities¶
- Hardware selection: A800 / 4090 / H20 / L40 / consumer vs pro GPU tradeoffs
- Inference stacks: vLLM · Transformers · Ollama · SGLang · TGI
- Model choice: open-source (Qwen · DeepSeek · Llama) · commercial APIs · AW36 proprietary
- Storage: Milvus · Qdrant · Chroma · pgvector · versioned indexes
- Networking: on-prem · half-on-prem · air-gapped · hybrid · data-egress detection
3 Representative Cases¶
Case 01 · Tier-3 Hospital AW36-72B + Doctor Assistant¶
- Problem: doctors spent 8-15 min looking up clinical guidelines / drug interactions / charting norms
- Stack: AW36-72B on 4×A800 · vLLM · Milvus · bge-large · BGE-reranker
- Boundary: all PHI stays on-prem · no public egress · internal audit log
- KPI: guideline lookup 8-15 min → 45s · 300+ daily-active doctors · discharge-summary drafting +50% efficiency · passed Level-3 protection compliance
- My role: architecture · hardware selection · vLLM productionization · intramural network integration · load testing · SLA
Case 02 · Air-Gapped Manufacturing Workshop Edge LLM¶
- Problem: air-gapped auto-parts workshop · 20+ years of expert knowledge locked in retiring foremen
- Stack: AW36-14B on single 4090 · Ollama · local vector DB · workshop tablet + PC dual endpoints
- Boundary: air-gapped · no public egress · offline knowledge sync
- KPI: incident triage time -45% · air-gapped available · digital knowledge base for future workers
- My role: hardware selection rationale (why 4090 over A800) · edge deployment scripts · on-site rollout
Case 03 · Enterprise Multi-Tenant LLM Gateway (20+ business lines)¶
- Problem: 20+ business lines with distinct AI needs · runaway token cost · uneven compliance posture
- Stack: OpenAI-compatible gateway · Redis metering · department quota · audit log
- Routing: on-prem AW36-72B (sensitive) + DeepSeek-V2 (general) + selective public APIs
- KPI: 20+ business lines shared · per-department token budget cap · group-wide compliance passed
- My role: gateway architecture · token metering · multi-tenant isolation · SLA
Delivery Discipline¶
- Sizing front-loaded: peak concurrency · p95 latency · data boundary · storage IO · failure accountability · cost caps — 6 axes quantified BEFORE hardware
- Inference stack fit: high throughput → vLLM · single-edge → Ollama · deep customization → hand-rolled Transformers
- Observability ready: token/sec · p50/p95/p99 latency · GPU util · VRAM fragmentation · failure rate — 5-panel dashboard
- Hot swap: model version · prompt version · index version — independent rollback
- Human backstop: high-risk paths (medical diagnostic support · legal final draft · financial decisions) preserve human review
Common Anti-Patterns¶
- ❌ 8×A800 cluster for 5 QPS — should be single 4090
- ❌ vLLM without warm cache — first request 30s
- ❌ 72B deployed but using bge-small — RAG foundation cracked
- ❌ on-prem deploy but audit log leaves premises — compliance blown
Related¶
- Private AI deployment sizing rules (6 steps)
- Full 20-case portfolio · PDF
- Vendor acceptance checklist (8 hard KPIs)
- Related directions: RAG knowledge bases · Multi-model routing