Eval cases & datasets

A dataset is how you tell Donkit what "good" means for your agent. Each case captures a representative input and the expected outcome. For Q&A-style cases that typically means the query, the expected answer, optionally the tool you expect the agent to use, and — for knowledge agents — the source passage the answer should be grounded in.

Working with datasets

• Create a dataset for an agent, give it a name and description, and add cases — by hand or by generating them (below).
• Generate a dataset automatically. If your agent has a knowledge base, Donkit can read your uploaded documents and draft question/answer cases from them, so you start with a realistic test set instead of a blank page. You choose how many cases to generate.
• Attach judges. Each dataset carries the set of quality checks used to score it. If you don't configure them, Donkit picks sensible defaults based on the kinds of cases in the dataset — for example, adding a grounding/hallucination check when cases reference source documents.
• Run an experiment. Running a dataset against your agent's current version dispatches an evaluation and streams live progress. Each run is saved with its score.
• Edit and re-run. Datasets are living assets — add cases as you discover new edge cases, then re-run to confirm a fix.

Start from your documents

The fastest way to a useful dataset is to generate cases from a knowledge base you've already connected. Review and trim the generated cases, then keep adding the real questions your users ask.

What you get

A reusable, versioned "exam" for your agent. Every run records how many cases passed, the overall score, and per-case detail. Because runs are cached by dataset and configuration, re-running an unchanged setup is fast, and comparing two runs shows exactly what changed.

Next: see the qualities each case is scored on in Dimensions & metrics.