AI Vendor Acceptance Checklist¶
TL;DR: 8 hard criteria. Miss any one → don't sign. The critical three: contract-quantified criteria + client-owned retest set + documented data boundary.
1. Contract Specifies Quantified Pass Criteria¶
Bad: "System can answer user questions, subject to mutual confirmation."
Good: "20 real buyer questions, tested across GPT-4 / Claude / Doubao, ≥ 70% citation hit rate on delivery, ≥ 60% on 14-day retest = pass."
Key: criteria must be quantifiable, retestable, and shared between both parties.
2. Retest Set Owned by Client (Not Vendor)¶
Vendor-owned "golden sets" are conflict of interest.
Correct process: - Client provides 20 real buyer questions (from actual business — not fabricated) - Client keeps expected answers - Vendor never sees expected answers, only produces model output - Compare together
3. Data Boundary Documented in Writing¶
Contract appendix must include:
| Data type | Leaves premises? | Allowed processing location | Retention |
|---|---|---|---|
| Customer PII (SSN / phone) | No | Client internal network | Legal minimum |
| Business data (orders / inventory) | No | Client VPC or internal | ≤ 30 days |
| Input logs (prompts) | Depends on tier | Private API gateway | ≤ 7 days |
| Output logs (completions) | Depends | Private API gateway | ≤ 7 days |
| Anonymized analytics (token counts) | Yes | Vendor-side | ≤ 90 days |
No table = no signature.
4. Cost Accounting Down to User / API-key Level¶
Vendor's monthly bill must support breakdown by:
- User (which internal team consumes most)
- API key (which product line consumes most)
- Model (which model is costliest)
- Time period (peak-hour cost)
Total-bill-only vendors = you're blind when problems hit.
5. SLA with Breach Penalties¶
Common metrics + penalties:
| Metric | Threshold | Penalty |
|---|---|---|
| Service availability | ≥ 99.5% | Hourly credit |
| Output error rate | < 3% | 30% refund if cumulative > 5% |
| Data leak | 0 | Full refund + disclosure |
| Verifiable factual error rate | < 2% | Per-incident $1K credit (example) |
SLA without penalties = worthless paper.
6. Model + Prompt Versioning with A/B Toggle¶
Vendor must demonstrate:
- Prompts version-controlled (Git, client can review)
- Model abstraction layer (no hardcoding, A/B switchable)
- Canary rollout (5% → 20% → 50% → 100%)
- Rollback switch (1-minute revert)
At demo: ask them to live-switch a prompt version. Can't? Not production-ready.
7. Client-Accessible Dashboard¶
Not vendor-internal only:
- Cost dashboard (client sees real-time consumption)
- Hallucination rate (daily sampling)
- User satisfaction (optional NPS)
- System health (availability)
No client dashboard = you trust vendor's "everything's fine".
8. Exit Clauses¶
Contract must specify:
- Data: within 30 days after service ends, client can bulk export
- Model: client-tuned LoRA / weights belong to client
- Prompts: version history of client-used prompts belongs to client
- Logs: client can export all logs
- Knowledge base: client-uploaded KB belongs to client
No exit clause = your data is hostage.
Print-Ready Audit Table¶
Tick each item live during vendor demo:
- [ ] Pass criteria in contract, quantified
- [ ] Retest set with me, vendor hasn't seen
- [ ] Data boundary documented, tier-based
- [ ] Cost accounting granular to user / API key
- [ ] SLA with breach penalties
- [ ] Model + prompt versioning A/B (demonstrated live)
- [ ] Dashboard client-accessible (log in live)
- [ ] Exit clauses cover data / model / prompt / logs
8/8 → sign 6-7/8 → negotiate addenda ≤ 5/8 → walk away