> ## Documentation Index > Fetch the complete documentation index at: https://docs.tied.com/llms.txt > Use this file to discover all available pages before exploring further. # Evaluation > Knowing whether a change made the agent better or worse — before your customers find out. Every agent vendor ships changes. The question is whether they can tell you what a change did. Tied runs an evaluation suite on every published version, gates the publish on the result, and keeps the history so you can compare two versions rather than compare a version against a memory. ## Golden sets A golden set is a list of real questions with the answers you have agreed are correct. You start with one Tied assembles from your own traffic and your knowledge base. You add to it constantly, and mostly not on purpose: **every case worked through the [quality queue](/agents/operations) lands in a golden set**. A resolution that went wrong once becomes a permanent regression test, so it cannot go wrong the same way twice. **The golden set is the most valuable thing you own here.** Knowledge can be re-crawled and prompts can be rewritten, but a curated set of real questions with agreed answers is accumulated and cannot be reconstructed. It is exported on request. ## What gets measured Retrieval quality — **recall\@5** and **MRR** against a labelled set. Measured separately from answer quality, because "didn't know" and "couldn't find what it knew" have opposite fixes. Graded against the golden set. Correct, incomplete, or wrong — with wrong broken out from incomplete, because they carry very different costs. Whether the answer is actually supported by the sources retrieved, rather than fluent and unsupported. A confident answer to a question the knowledge base does not cover is the failure that matters most, and it is the one a confidence score cannot catch — a model confabulating an answer is not uncertain about it. Every refusal category and every escalation ground has its own cases. This is checked in both directions: escalating when it should not is a real regression, not a safe default. Multi-turn replay against a simulated customer — did it ask before it wrote, did it escalate at the right turn, did it silently retry a financial action. Single-turn scoring cannot see any of this. Adversarial probing on every build: jailbreaks, prompt injection, and off-topic misuse. ## The publish gate The same rule as the [studio](/agents/studio), stated where it is enforced: | Change | Gate | | - | - | | **Knowledge-only** | Score shown, explicit acknowledgment required, publish allowed. | | **Anything that can write** | The golden set must pass. Every plan. | | **Enterprise** | Always gated. | **Blast radius decides, not revenue.** A write against a customer's real order is the same risk on the cheapest plan as on the largest contract, so it gets the same gate. ## Online scoring is sampled A share of live conversations is scored continuously, so quality drift shows up before it shows up in your CSAT. It is **sampled rather than applied to everything**, deliberately. Scoring every trace would compete for budget with the gate that blocks bad releases, and of the two, the gate matters more. The sample rate is a setting Tied can turn down without a deploy. ## Reported numbers reconcile against your invoice The agent's reported resolution rate counts **confirmed resolutions only** — the same definition billing uses — alongside explicit escalation and abandonment accounting. **So the rate you are shown is auditable, line by line, against what you were charged.** Because billing and reporting share one definition, any divergence between the two is a bug in one of them, not a difference of methodology. A resolution rate that cannot be reconciled against the invoice becomes the thing a customer disputes instead of the thing that builds trust. Escalations and abandonments are reported in full rather than quietly excluded from a denominator. ## Learning from what actually happened Every confirmed resolution, every human correction, and every approved flow is training signal, not just retrieval content. Two surfaces: | Corpus | Trained on | Used by | | - | - | - | | **Pooled** | Anonymised, aggregated data across accounts, with personal data stripped | Everyone | | **Private layer** | One account's own data, never crossing to another | That account only, on top of the pooled corpus | **The pooled corpus is anonymised-tier only.** Models memorise rare strings, and a customer's name in a support transcript is exactly the shape of thing that gets memorised — so raw data is never pooled, on any plan, under any setting. Raw data is used only in an account's own private layer. Self-serve accounts contribute to the pooled corpus by default, disclosed in the terms and the DPA at signup rather than discovered later. Enterprise accounts are isolated by default and may opt in. There is a hard exclusion setting that means never. **Withdrawal applies forward, not backward.** A trained model cannot be un-trained, so withdrawing means exclusion from the next training run. That is stated plainly here because implying otherwise would be false. ## Next The quality queue that feeds the golden sets. Where a version is published and rolled back. What the adversarial probes are probing. The definition that reporting and billing share. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.