Dimensions & metrics¶
Dimensions are the specific qualities your agent is scored on. Each is scored 0–100% by an automated judge, then combined into a weighted overall score with a written explanation. Judges apply a configurable pass threshold, and you can weight dimensions to match what matters for your use case.
Q&A / dataset dimensions¶
Applied to single-response cases:
| Dimension | What it checks |
|---|---|
| Correctness | Is the answer right versus the expected answer. |
| Hallucination / groundedness | Does the answer stick to real source material rather than inventing facts. |
| Answer relevance | Does it actually address the question. |
| Conciseness | Is it appropriately brief. |
| Plan adherence | Did the agent follow the expected approach. |
| Toxicity | Is the output safe and appropriate. |
| Code checker | For agents that produce code, does the code hold up. |
Dataset runs also report supporting signals: success rate, retrieved-source match rate, expected-tool match rate, average tools called, and average per-case latency.
Dialogue dimensions¶
Applied across simulated multi-turn conversations, grouped so results are easy to act on:
- Goal completion — did the agent accomplish what the user actually wanted by the end.
- Response quality — relevance of each reply; whether it acts versus merely narrating; context consistency (no self-contradiction across turns) and context retention (recalling facts from earlier turns).
- Knowledge / RAG quality — groundedness and answer match against your knowledge base, plus retrieval quality.
- Tool use — tool accuracy (the right tool), tool necessity (called when needed, skipped when not), redundancy/efficiency (no wasteful or repeated calls), and detection of tool loops.
- Safety & policy — security compliance, memory safety, intent/injection detection, whether the agent asks for confirmation before risky actions, and how it handles ambiguous requests.
- Planning — task decomposition, goal adherence, and plan safety.
The dialogue overall score weights the highest-signal dimensions most heavily (response relevance, tool accuracy, and context retention), factors in knowledge-grounding for RAG agents, and adjusts downward when safety, memory, or planning problems appear.
What you get¶
Instead of one opaque number, you see which qualities are strong and which are weak — each with a threshold and an explanation. For the weakest areas, Donkit surfaces a short list of automatically identified "weak spots" (for example, "a required tool was skipped in 4 of 5 scenarios" or "relevance drops after turn 4") that tell you exactly what to fix.