Tracing

When an agent gives a wrong answer, the final output doesn't tell you why. Was it the search, the tool call, the prompt, or the model? Without traces you end up guessing and re-running the agent by hand.

In this workshop we take the FAQ agent from our earlier AI Shipping Labs workshops, instrument it with OpenTelemetry and use the traces to understand what it actually did.

What we'll cover: - Why logs and the final answer are not enough to debug an agent - Instrument the agent with OpenTelemetry so every LLM call and tool call becomes a span in one trace - Send the traces to Langfuse (open source, can be self-hosted) and read them: inputs, outputs, latency, tokens and cost per step - The same OpenTelemetry traces work with other backends too - Use a trace to find why one specific answer went wrong - Turn interesting traces into examples for evals

You'll leave with a working tracing setup you can add to your own agent project.

Who it's for: people who already have an agent or RAG app (or followed one of our earlier workshops) and want to see what happens inside it.

Prerequisites: Python, an OpenAI (or other LLM provider) API key, a basic agent with tools. The code and recording are shared with members after the session.