Personal RAG Coach
An AI strategist for brands building a presence online. It takes the teaching of experts a brand already trusts and turns it into a coach that knows the business, answers from that material, and writes the scripts.
The problem
There's no shortage of advice online. The hard part is knowing which advice to follow and how it applies to your business. Most brands find a few teachers they trust, take the course, and the knowledge ends up sitting in a folder.
This coach turns that material into an asset that keeps working. It learns the brand's profile, audience and past decisions, answers from the sources it was given, and says so when those sources don't cover a question.
What a brand gets
- Strategy on demand, grounded in experts the brand chose, not the whole internet.
- Advice that fits. It remembers the business, its beliefs and every correction.
- Content, not just advice. Strategy becomes briefs, outlines and ready-to-record scripts.
- Honest limits. When the material doesn't cover something, it says so instead of guessing.
One chat turn
- The message streams in. The coach loads the brand's profile, recent turns, memories and open drafts.
- An agent picks from about 50 tools: knowledge search, script pipelines, revisions, memory.
- Every model call goes through one gateway that tracks cost and cache hits.
- Before the reply ships, guards check it for completeness and unsupported claims.
By the numbers
- Script time cut from 786 s to 86–105 s, with matching quality on three of four rubrics.
- Retrieval hit@5 of 0.909 on 33 labeled questions (hit@1 0.727, MRR 0.80). Small sample, real number.
- About 50 agent tools, 4 model tiers, at most 10 tool rounds per turn.
Click any box for what it does, why it matters, how it works and what it cost. Filter by skill, or press a ▶ walkthrough.
Which model does what
| Model | Does |
|---|---|
| Opus 4.8 | Chat loop, strategy, briefs, humanizer drafts |
| Fable 5 | Outline and full script in one call. Opus retries if it refuses. |
| Sonnet | Tagging, query routing, ideas, hooks, revision patches |
| Haiku 4.5 | Answer and fabrication guards, query rewrites, summaries, fallback |
| Voyage | Embeddings and reranking |
Opus runs the chat because the chat model both routes the request and writes what the user reads. Head to head, Opus with adaptive thinking tied Fable 6–6 at roughly 55–60% of the cost.
Retrieval
- Haiku rewrites a follow-up into a standalone question.
- Sonnet picks the question type and filters while the query is embedded.
- A small TF-IDF + SVM classifier sets the search budget. Haiku is asked only when it's unsure.
- Hybrid search in Qdrant: dense vectors plus BM25, merged by rank fusion.
- Voyage reranks, with caps of 2 chunks per file and 3 per source.
- Below a relevance floor, the coach says it has no good material instead of guessing.
Guardrails
- Agentic outside, deterministic inside. V1 routed every request to one of 12 fixed pipelines, and planning requests lost their grounding. V2 made the pipelines tools the model calls, and code controls each step inside them.
- Models never choose IDs. A model once returned
section_5for a real ID, and revisions silently skipped sections. Now code mints every ID and checks every reference. - Score each option, let code pick. Asking a model for "the best one" gave different picks each run. Scoring options one at a time and taking the max in code made the choice repeatable.
- One retry, then fail loudly. Model reviewers agreed with themselves about 47% of the time, and repeated fix passes made scripts worse.
- Fail closed on writes, open on advice. A judge outage never blocks a reply. A bad write never lands.
- Structured output through one helper. A typed schema becomes a forced tool call, and invalid output is repaired.
Evals
A free replay harness: an agent answers as the model, blind judges score, and network calls are blocked. It preferred the new version in 8 of 9 scenarios. It's a proxy, not a release certificate.
Stack
| Layer | What |
|---|---|
| API | Python 3.12, FastAPI, streaming over SSE, Pydantic 2 |
| Vector store | Qdrant with dense Voyage vectors (1024-d) plus a BM25 sparse vector |
| Models | Anthropic SDK through one gateway with a hook bus and call ledger |
| Reranking | Voyage Rerank 2.5 |
Key decisions
- Collapse the chain. A staged script chain took 786 s. One call plus free code checks takes 86 to 105 s.
- No direct provider calls. A CI check forbids them, so cost, caching and audit all go through the gateway.
- Cache-stable prompts. System prompts stay byte-identical, and a hook flags anything that breaks the cache.
- Freeze flags per turn. A flag flipped mid-answer can't change behavior halfway through.
- Embedding hygiene. Documents and queries use the right embedding mode. The wrong one drops retrieval by about 20%.