Direction 4 · Multi-Model Routing¶

Multi-model routing = region + cost + sensitivity + quality four-axis routing — not "hook up a few APIs". 3 representative cases + delivery discipline.

Core Capabilities¶

• Routing: sensitivity tiers · region residency · cost budget · quality preference · fallback
• Gateway: OpenAI-compatible unified entry · litellm · portkey · custom
• Observability: cost · latency · failure rate · engine distribution · department quota
• Fallback: primary → backup → cache → human review
• Evaluation: per-prompt A/B · quality scoring · cost comparison

3 Representative Cases¶

Case 01 · Cross-Border SaaS Triple Gateway (Region / Cost / Sensitivity)¶

• Problem: SaaS users in US / EU / CN · residency compliance varies · cost runaway
• Stack: triple gateway (OpenAI US / Anthropic EU / AW36 CN on-prem) · sensitivity tiers · cost budget
• Strategy: US users default OpenAI · EU users default Anthropic · CN users default AW36 · sensitive data forced local
• KPI: per-token cost -38% · 100% residency compliance · user latency -20%
• My role: gateway architecture · routing strategy · cost model

Case 02 · Hospital Public/Private Hybrid Routing (Sensitive → On-Prem, General → Public)¶

• Problem: hospital compliance forbids PHI on public net, but full on-prem GPU is expensive
• Stack: PHI detection + DLP at gateway · on-prem AW36-72B · public GPT-4o
• Strategy: contains PHI → forced on-prem · non-PHI and non-sensitive → public · all traffic redacted audit
• KPI: total cost -46% vs full on-prem · meets hospital compliance · no PHI egress records
• My role: routing architecture · PHI detection rules · compliance assessment

Case 03 · Content Factory Multi-Model A/B (Best-per-Prompt)¶

• Problem: Chinese content marketing agency · prompt types perform differently on different models
• Stack: per-prompt A/B across GPT-4o / Claude 3.5 / AW36-Max / DeepSeek in parallel · auto-selection · human final review
• Strategy: head prompts A/B · long-tail routed to cheapest · human review retained
• KPI: client conversion +21% · per-piece cost -30% · human review workload -50%
• My role: A/B platform · scoring model · client integration

Delivery Discipline¶

1. Sensitivity tiers front-loaded: user data · commercial secrets · public info — three tiers with hard rules
2. Region residency compliance: EU GDPR · China data-export · US CCPA — every one has hard boundaries
3. Cost budget three-layer: per-call cap + per-user daily budget + global circuit breaker
4. Fallback path: primary → backup → cache → human review — four-tier degradation
5. Observability by department: department · user · prompt type · engine — four-axis crosscut

Common Anti-Patterns¶

• ❌ Just "hook up a few APIs" from day one, no routing strategy, cost blows up
• ❌ Only look at total cost, not per-prompt-type value/cost — long-tail still on GPT-4o
• ❌ No fallback path — primary 429 kills business
• ❌ Observability at aggregate level only — can't tell which department/user is burning cash

Related¶

• Full 20-case portfolio · PDF
• Related directions: Private LLM deployment · AI+DevOps

Next: Download PDF · 30-min multi-model architecture alignment