LLM evaluation,
with a record an auditor can read.

• Nine metrics: relevancy, faithfulness, hallucination, correctness, bias, toxicity, task completion, tool correctness and MCP use
• Works with Anthropic, OpenAI and OpenAI-compatible endpoints
• An LLM judge scores each answer and explains the score
• Each run stored as evidence against AI governance controls

LLM evaluation is testing a language-model application against a fixed set of prompts and scoring the answers on the properties that matter to you: whether they are relevant, grounded in the source, correct, free of bias and toxicity, and for agents whether the right tools were called with the right arguments. Run on every model or prompt change, it is the regression test for behaviour that unit tests cannot see.

For governance the record matters as much as the score. ISO/IEC 42001 and the EU AI Act both expect testing to be planned, repeated and documented; a dated run with its inputs and results is that documentation.

1. Connect your modelPoint the platform at your model through the Anthropic API, the OpenAI API or any OpenAI-compatible endpoint, including self-hosted models that expose one.
2. Write test casesEach test case has an input, and optionally an expected answer, the retrieval context the answer should rest on, and the tools or MCP servers the agent may use.
3. Choose metrics and a thresholdPick the metrics that matter for the case. Each is scored from 0 to 1, higher is better, and the default pass mark is 0.7, which you can change per test case.
4. Run and judgeThe platform calls your model with the input, then a judge model scores the answer on each metric and gives a one-sentence reason for the score.
5. Keep the recordEvery run is stored with its input, output, scores, reasons and date, so you can compare runs over time and show the history to an auditor.

Because an agent can give a fluent answer while calling the wrong tool, or the right tool with the wrong arguments. Tool correctness and MCP use are scored on their own, so a regression in how the agent acts shows up even when its words still read well.

A judge model is used because the properties that matter, such as relevance, grounding and tone, cannot be checked with an exact string match. A second model reads the input, the answer and any context, and scores it against a written rubric for one metric at a time.

A judge can be wrong, so every score comes with its reason. A person can read why an answer scored 0.4 on faithfulness and decide whether the judge was right. Treat the scores as a regression signal across runs rather than a single verdict on the model.

Both expect testing to be planned, repeated and recorded. ISO/IEC 42001 asks an organisation to operate its AI systems under controlled processes and to verify and validate them, with documented evidence of the results. For high-risk systems, the EU AI Act requires a risk management system (Article 9), appropriate levels of accuracy and robustness (Article 15) and a quality management system that covers testing and validation (Article 17). The NIST AI RMF calls this work the Measure function.

None of these frameworks names a metric or a pass mark. What an assessor looks for is a defined test set, criteria agreed in advance, results kept over time, and action when results fall. A dated run with its inputs, scores and reasons is that record. See ISO 42001, the EU AI Act and NIST AI RMF.

Run evaluations whenever something that shapes the answer changes: a new model version, a changed system prompt, a new retrieval index or a new tool. Those are the changes that quietly alter behaviour without breaking any unit test.

Keep the test set stable between runs so scores are comparable, and add a case every time a real failure is found in production. Over time the test set becomes a record of every way the system has gone wrong, and a guard against it happening again.

Test an agent's actions separately from its words. An agent can write a fluent summary after calling the wrong tool, or the right tool with the wrong arguments. Tool correctness scores the calls themselves, and MCP use scores whether the agent used the tools, resources and prompts its MCP servers offered.

Record the tool calls with each run. When a score drops, the calls show what the agent did differently, which is usually faster to diagnose than the text. The wider governance work, such as inventory, risk classification and oversight, is covered in AI governance.

• The inventory of AI systems and which ones are tested
• The test plan: what is tested, how often and against what criteria
• Results over time, not a single run before the audit
• What happened when results fell below the agreed mark
• Who reviewed the results and approved the release
• How test cases are added when real failures are found

• It does not decide your pass marks; you set the threshold per test case
• It does not certify a model or make an AI system compliant on its own
• It does not run without a judge model configured on the platform
• It does not generate your test set; test cases come from you
• It does not replace human review of high-risk outputs

Every result here is a control result on the same graph as the rest of the platform, so it reaches each framework that asks for it without being gathered again. Coverage lists the regimes.

Related reading: AI governance, the ISO 42001 guide and the EU AI Act guide.