| Item | Why it matters |
|---|---|
| Problem stated as a user outcome | Not "let's use AI". Name the user, the job, and what better looks like |
| Success metric defined and measurable | Decided before any build. "Accuracy above 85%" beats "improved experience" |
| Baseline captured | What does the current process deliver? Without this you cannot prove a lift |
| Build vs buy vs API decided | Rough cost per request, per month, per year. Even a bad estimate reveals the shape |
| Sponsor and users named | One accountable owner. Three named users who will test it |
| Item | Why it matters |
|---|---|
| Source data identified and reachable | If retrieval is needed, it must exist somewhere queryable |
| Rights and permissions confirmed | Who owns it, who can see it, what happens if it changes |
| Sensitive data classified | PII, PHI, financial, IP. Different handling for each |
| Residency and cross-border rules checked | Regulated industries stall here. Confirm before you build |
| Chunking or indexing strategy chosen | The unit of retrieval decides what the model can ever know. See QR-03 |
| Freshness plan defined | How stale is too stale? Who re-indexes, and when? |
| Item | Why it matters |
|---|---|
| Model selected by fit and cost | Route by workload, not by hype. See QR-02 and QR-15 |
| Prompt written to a spec | Role, task, context, constraints, output, escape hatch. See QR-01 |
| RAG or fine-tune decided, not both by default | Facts belong in retrieval, behaviour in fine-tuning. See QR-06 |
| Guardrails in place | Input filters, output checks, refusal patterns. Not optional |
| Prompt injection defence tested | Untrusted content fenced. Instructions inside data are ignored |
| Structured outputs where downstream code depends on them | JSON schemas, validation, retry on malformed |
| Item | Why it matters |
|---|---|
| Golden dataset built from real inputs | 50 to 200 hand-labelled examples. Not clean cases, real ones |
| Metric matches the goal | RAGAS for retrieval, exact match for extraction, LLM-as-judge for prose |
| Automated evals run in CI | Every prompt change and model change re-runs the set. See QR-11 |
| Adversarial and edge cases tested | Empty inputs, hostile inputs, injection attempts, off-topic requests |
| Fairness or bias checks where relevant | Especially for hiring, credit, health, legal. Document the results |
| Regression gate on the pipeline | Ships only when the golden set clears the bar |
| Item | Why it matters |
|---|---|
| Cost caps and rate limits set | Per user, per feature, per day. A runaway loop is expensive fast |
| Tracing and monitoring live from day one | Every prompt, every response, every tool call. Debug without guesswork |
| Prompt caching enabled where possible | Around 90% off repeated input if the front of the prompt is stable |
| Human-in-the-loop for destructive actions | Writes, deletes, sends. Friction is a feature, not a bug |
| Rollback path defined | Every deploy needs an undo. Prompts are code |
| Launch communications drafted | What it does, what it does not do, who to contact. Sets expectations |
| Item | Why it matters |
|---|---|
| Risk tier assessed | Against EU AI Act, NIST AI RMF, or your internal framework. See QR-13 |
| Decisions and limitations documented | A model card. What it does well, what it fails at, what data it saw |
| Data handling documented | What is logged, how long, who can access it, when it is deleted |
| Owner named for the monthly review | Someone is accountable. Not "the team", a person |
| Incident response runbook exists | What to do if it hallucinates in production, leaks data, or breaks |
| Retirement plan sketched | Every project ends. Decide up front how |
| The skip | Why it hurts later |
|---|---|
| Shipping without evals | Every regression is a surprise. Every rollback is a firefight |
| Chasing a demo, not an outcome | Impressive prototype, no real user problem, no path to production |
| Assuming the model can hold your knowledge | It cannot. Give it the documents (RAG). See QR-03 and QR-06 |
| One heroic prompt doing three jobs | Debug it once and you will split it. Split it up front instead |
| No cost cap, no rate limit | The first runaway loop teaches this the expensive way |
| "We will do governance later" | Later is after the incident. Do it now, or a regulator will |
| Testing on the happy path | The failure mode is what you did not test. Look for it deliberately |
| Picking the biggest model available | Route by workload. A 25x price gap makes routing worth engineering |