Evaluation and Monitoring

An agent that is not evaluated is a demo with a demo's lifespan: the next prompt change is a coin flip. The AISL material treats testing, monitoring, and evaluation as three separate disciplines — the buildcamp gives each its own week — and the workshops show the pipeline versions.

Testing agents

AI Engineering Buildcamp module 4 treats an agent like any other system under test: converting the notebook into a project, first agent tests, testing tool calls, fixtures and refactoring. The homework tests a DuckDB SQL agent. The point is that agents have contract behavior — which tools called with which arguments — and contracts can be tested. Structured Output is what makes model output contractual enough to test at all.

Solving an AI Engineer Home Assignment: Investment Coach Bot shows the same discipline under time pressure: test guidelines written first, real tests generated against them, then an evaluation harness that produces a score to move.

Evaluation: judges and synthetic data

Buildcamp module 6 goes beyond tests: offline evaluation, running the agent on scenarios, judge and alignment, and synthetic evaluation data — generating the user questions you did not think to test.

An Evaluation Framework with Batch and Flex Processing packages that into a pipeline around an unchanged agent: synthetic student questions in, agent runs, LLM-judge scores out, repeatable after every change. The pricing engineering is part of the lesson — the Batch API at roughly half price for generation and judging, Flex processing for the same rate in near-real time, prompt caching on top.

Jev for LLM Evaluations attacks the other half of that bill: the judge itself. Jev is a typed-decision model — it reads a piece of application state plus questions with named choices and returns a class per question, generating no output tokens. Ported onto the batch-and-flex pipeline, it judged the same ten-entry run for $0.000429 where the OpenAI judge cost $0.02439, and an open-weights alternative, Laya, scored the identical request body locally on a CPU. All three judges agreed almost everywhere on the sample. One trap is documented: packing several records into a single request returns one mixed answer, so the working pattern is one request per entry, run concurrently.

Monitoring in production

Buildcamp module 5 is the observability week: Logfire integrated with PydanticAI, span trees across runs, user feedback tracking, traces exported to Jaeger, and a DIY monitoring fallback that saves logs and events to SQLite.

Where they sit

Tests catch regressions, evaluation measures quality, monitoring watches the shipped thing. Guardrails — Guardrails — are the checks that run inside the request path.

Where to go next

• The agent under test: Agents
• The buildcamp weeks behind this page: AI Engineering Buildcamp