Back to workshops
Workshop Basic or above

Tracing a FAQ agent with Langfuse

November 4, 2026
Observability Tracing Evaluation Langfuse
Watch recording View code on GitHub

Recording available with Main or above.

We add Langfuse tracing to a FAQ agent, then turn its traces into reviewed evaluation examples. We examine model calls, search results, token usage and cost before choosing which answers belong in a dataset.

We capture the agent's calls automatically, but review its answers before creating reference answers.

We start with the FAQ application from End-to-End Agent Deployment. We already use it to search the DataTalks.Club FAQ and stream answers through FastAPI. We use OpenCode to add instrumentation without rewriting that application.

We first try manual OpenTelemetry spans, then simplify the integration with Langfuse's OpenAI client and @observe decorators. After checking the traces, we create a human annotation queue and promote reviewed answers into a dataset.

We use these resources throughout the walkthrough:

Basic or above required

Upgrade to Basic to access this workshop

  • Full workshop overview
  • Complete page list
  • Step-by-step tutorial
  • Cancel anytime
Upgrade

Tutorial pages

  1. Part 1: Prepare the FAQ agent
  2. Part 2: Trace model calls and tools
  3. Part 3: Use a local Langfuse instance
  4. Part 4: Review answers before creating a dataset
  5. Part 5: Promote reviewed traces to evaluation examples
  6. Next steps and file reference

Related content

Workshop Basic or above

Jev for LLM Evaluations

In the batch and flex workshop we built an eval for the FAQ agent and scored it with an LLM judge. Here we swap that judge for Jev, a model that returns typed decisions instead of…

September 22, 2026