Skip to content

Evaluation overview

Donkit treats agent quality as something you measure, not guess. Before you publish, and continuously after you go live, the platform runs your agent against realistic inputs, scores its behavior on quality dimensions, and shows you where it's weak — so you can prove your agent works and keep improving it. This is the foundation of Donkit's self-optimizing promise: you can't optimize what you don't measure.

The mental model

Evaluation is built from four ideas:

  • Build quality & task checks — after every build, Donkit grades the result and tries a few real tasks on the agent. See Build quality & task checks.
  • Eval cases & datasets — the examples that define "good." A case is an input (a question, or a multi-turn goal) plus what a correct response should look like. A dataset is a named collection of cases you run repeatedly. See Eval cases & datasets.
  • Dimensions & metrics — the specific qualities each response is scored on (accuracy, groundedness, tool use, safety, and more), each scored 0–100%. See Dimensions & metrics.
  • Scores — per-dimension scores plus a weighted overall score, each with a plain-language explanation of why.
  • Runs (experiments) — one execution of a dataset or scenario set against a specific version of your agent. Runs are saved, so you can compare versions over time.

The kinds of evaluation

You reach these under your agent's Evaluate tab. Availability depends on your plan.

  • Pre-release evaluation

    Test before you ship. Run your agent against a dataset of question/answer cases, or simulate full multi-turn conversations with a realistic virtual user to surface failures that only appear across a dialogue.

    Cases & datasets

  • Adversarial testing

    Actively try to break your agent — off-topic requests, boundary-pushing, prompt-injection and safety probes.

    Adversarial testing

  • Live monitoring

    Continuously score real conversations after launch, so quality regressions surface in production.

    Production scores

What you get

A clear, versioned quality picture of your agent: an overall score, a per-dimension breakdown, concrete failing examples you can click into, and a side-by-side comparison against the previous version. These are the inputs you need to decide whether an agent is ready — and where to invest next.

The through-line across everything below: define quality (datasets, dimensions) → prove it (pre-release, adversarial) → keep it (live monitoring) → improve it (optimization) — with saved, comparable runs at every step.