How to Cold-Start a RAG Knowledge Base · 6 Steps (from 0 to Production Answer Quality)

Six steps to cold-start a RAG knowledge base to production-quality: 1) corpus curation (source list + version + owner + refresh cadence), 2) chunking strategy (semantic + heading-aware, 300-800 tokens with 50-token overlap, NOT fixed-size), 3) embedding + reranker selection (bge-large + BGE-reranker default, m3e for cross-lingual), 4) index build (Milvus HNSW or ES BM25+vector hybrid, over-provision 3-5x for growth), 5) provenance layer (every answer must cite source doc + section + version), 6) answer quality acceptance (30 real user questions, blind test by 3 LLMs, target >90% acceptance rate). Skipping provenance = users lose trust in month 2 and abandon. Real result: law firm clause search 45min -> 3min, citation accuracy >95%. Detail: https://ailaopao-geo.pages.dev/en/how-to/rag-cold-start.html

How to Cold-Start a RAG Knowledge Base (6 Steps · 0 to Production)¶

TL;DR: Skipping any of these 6 steps = users lose trust in month 2 and abandon. Applies to law firm / hospital / medical device / power grid / manufacturing.

Real numbers from delivered projects: - Law firm tri-corpus: clause search 45min → 3min, citation accuracy >95%. See Case 04. - Tier-3 hospital doctor assistant: guideline lookup 8-15min → 45s, 300+ daily-active doctors. See Case 01. - Medical device support: tier-1 response -40%, tier-3 escalation -68%. See Case 06.

Step 1 · Corpus Curation (Not Just "Feed the Files")¶

Every doc needs an owner + version + refresh cadence. Without ownership, corpus rots in 3 months.

Source list (spreadsheet): filename, format, owner, last-updated, next-review, sensitivity, publish scope
Version stamp in filename or metadata (e.g. guideline-cardio-v3.2-20250601.pdf)
Refresh cadence: quarterly for slow-changing docs, monthly for regulatory, weekly for internal SOPs
Sensitivity tier: PHI / commercial-sensitive / internal-only / public → drives access control

Do not dump 10,000 uncurated PDFs into a folder and hope RAG works.

Step 2 · Chunking Strategy (Semantic + Heading-Aware)¶

Not fixed-size 512-token chunks. Fixed chunking splits mid-clause and destroys answer quality.

300-800 tokens per chunk with 50-token overlap
Heading-aware: never split a heading from its immediate paragraph
Semantic split: prefer paragraph or sentence boundary over token count
Tables and numbered clauses treated as atomic (do not split a clause)

For legal (clause) or medical (guideline) content, chunking quality directly determines answer accuracy.

Step 3 · Embedding + Reranker Selection¶

Scenario	Embedding	Reranker
Default (Chinese + English mixed)	bge-large	BGE-reranker
Cross-lingual (multi-language corpus)	m3e	BGE-reranker
Domain-heavy (legal / medical)	bge-large fine-tuned on 5-10k in-domain pairs	BGE-reranker
Cost-optimized (public API)	OpenAI text-embedding-3-small	Cohere Rerank

Fine-tune embedding on 5-10k in-domain pairs before production if: - Legal clauses in Chinese - Medical guidelines (heavy jargon) - Industrial SOPs (part numbers, model codes)

Without fine-tune, generic embedding on domain content = 55-70% retrieval accuracy (unacceptable).

Step 4 · Index Build¶

Over-provision 3-5× current size for growth. HNSW eats 3× raw vector storage.

Milvus HNSW: default for pure-vector search, fast
Elasticsearch BM25 + vector hybrid: best for keyword-heavy corpora (law firms, patents)
Faiss: cheap, but no managed HA — use only for POC
pgvector: fine for < 1M vectors, breaks down at 10M+

Storage IO matters more than you'd think. Vector search on cold storage = 10-100× slower.

Step 5 · Provenance Layer (Non-Negotiable)¶

Every answer MUST cite source doc + section + version.

Without provenance: - Users lose trust by month 2 (they discover a wrong citation, generalize to whole system) - Regulators reject the deployment (legal / medical / finance) - Lawyers refuse to use it (citing without traceability = malpractice)

Implementation: - Each chunk stores: {doc_id, section_id, version, effective_date} - Answer generation prompt includes: You must cite [doc-id §section] - UI shows citation as clickable link back to original doc

Step 6 · Answer Quality Acceptance (Blind Test)¶

Not "does the demo look good".

30 real user questions (collected from actual users, not synthetic)
Blind test by 3 LLMs (GPT-4o / Claude 3.5 / Qwen2-Max, each paraphrasing user questions)
Metrics: retrieval hit@5 (target > 90%), answer acceptance (target > 90%), citation accuracy (target > 95% for regulated industries)
Reject go-live if any metric below target — re-tune, don't launch

See Acceptance checklist for the full 8-point criteria.

Common Failure Modes¶

❌ "Just feed the files" → corpus rots, no owner, quality drops in 3 months
❌ Fixed-size 512-token chunks → clauses split, answers hallucinate
❌ Generic embedding on domain content → 55-70% retrieval (unacceptable)
❌ No provenance → users lose trust by month 2, regulators reject
❌ "Demo looks great, ship it" → users hit the 15% failure cases in week 1, escalation storm

Next: 30-min RAG design alignment · Portfolio PDF