Course External course

AI Engineering Buildcamp: From RAG to Agents

Alexey Grigorev

AI/ML engineer, founder of AI Shipping Labs and DataTalks.Club.

ai-agents rag llm testing monitoring evaluation

A hands-on course on building production AI applications with large language models: retrieval-augmented generation (RAG), structured output, AI agents, testing, monitoring, and evaluation. Learners build real projects -- a documentation assistant, a YouTube summarizer, a books RAG system, and their own capstone -- across seven core modules, from foundational RAG through agentic flows to production-grade testing, monitoring, and evaluation.

Current cohort

Cohort 4

Started September 21, 2026

Syllabus

Cohort 4 schedule · Preview · Times shown in Europe/Berlin

Pre-work and logistics

Course Logistics 5 lessons

Course Overview Communication Homework and Project Submissions Learning in Public Office Hours

Main content

Week 1 · Foundations: LLMs, RAG, and Structured Output Sep 21–27 4 topics · 27 lessons Next due Oct 6, 2026 01:59 Europe/Berlin · 2 deadlines

Session 1 Week 1 Overview

Environment Preparation 6 lessons

GitHub Codespaces Windows Linux MacOS AI-Assisted Development Libraries Overview

Foundation: LLMs, RAG and Search 8 lessons

Section Overview Documentation Agent: Running Example for the Course OpenAI API RAG (Retrieval-Augmented Generation) Getting Documentation Data from GitHub Indexing Data for Search with minsearch Chunking Documents for Better Retrieval End-to-End Document Assistant

Structured Output 3 lessons

Structured Output Structured Output for Documentation RAG Streaming Structured Output

Project Work 5 lessons

Project Work From Idea to Submission Project Ideas for Your AI Capstone Cohort 2 Project Ideas Other Project Ideas

Module 1 Homework: Document Processing with AI Due Oct 6, 2026 01:59 Europe/Berlin Homework Module 1 Capstone: Your AI Project Due Oct 6, 2026 01:59 Europe/Berlin Homework Optional Content Optional

Week 2 · RAG in Practice: Use Cases and Technologies Sep 28–Oct 4 1 topic · 4 lessons

Session 2

RAG Use-Cases Overview 2 lessons

Week overview Reusable RAG Class

Optional Content Optional

Week 3 · Agentic Flows Oct 5–11 2 topics · 17 lessons Next due Oct 13, 2026 01:59 Europe/Berlin · 2 deadlines

Session 3

Agentic RAG 6 lessons

Agent Basics Overview Agentic RAG with Tool Calling Agentic Loop Q&A Loop and an Agent Library Visual Summary Structured Output for Agents (Optional) Optional

Agent Frameworks 6 lessons

Agent Frameworks Overview ToyAIKit for Agent Implementation Organizing Tools in a Class Agentic Search: Going Beyond RAG Pydantic AI: Production-Ready Agents Event Stream Handlers

Week 3 Homework: Building a Wikipedia Agent Due Oct 13, 2026 01:59 Europe/Berlin Homework Week 3 Capstone: Adding Agents to Your Project Due Oct 13, 2026 01:59 Europe/Berlin Homework Optional Content Optional Previous Week 3 Overview Optional

Week 4 · Testing AI Applications Oct 12–18 3 topics · 22 lessons Next due Oct 20, 2026 01:59 Europe/Berlin · 2 deadlines

Session 4 Testing and Evaluation Overview

Project Structure 6 lessons

Preparing the Project Converting a Jupyter Notebook into a Python Project Streaming with PydanticAI Optional Streaming Methods in PydanticAI Optional Structured Streaming with PydanticAI Optional Streamlit Frontend for Documentation Agent Optional

Testing for Agents 6 lessons

Testing for Agents First Agent Test Testing Tool Calls Fixtures and Refactoring More Tests: Code Blocks and Adding References Tracking Costs Optional

Test Judges 5 lessons

LLM Judges Creating an LLM Judge From Usage to Test Cases LLM Judge Tests Improving Judge Tests

Week 4 Homework: Testing a DuckDB SQL Agent Due Oct 20, 2026 01:59 Europe/Berlin Homework Week 4 Capstone: Testing Your Agent Due Oct 20, 2026 01:59 Europe/Berlin Homework Optional Content Optional

Week 5 · Monitoring Oct 19–25 1 topic · 10 lessons Next due Oct 27, 2026 00:59 Europe/Berlin · 2 deadlines

Session 5

Monitoring with Pydantic Logfire 5 lessons

Logfire Overview Logfire Integration with PydanticAI Joining Multiple Runs Under One Span Tree Tracking User Feedback Downloading Traces Data from Logfire

Week 5 Homework: Monitoring a Trivia Agent with Logfire Due Oct 27, 2026 00:59 Europe/Berlin Homework Week 5 Capstone: Monitoring Your Agent Due Oct 27, 2026 00:59 Europe/Berlin Homework Optional Content Optional Previous Week 5 Overview Optional

Week 6 · Evaluation Oct 26–Nov 1 2 topics · 15 lessons Next due Nov 3, 2026 00:59 Europe/Berlin · 2 deadlines

Session 6 Overview

Measure and Align: Offline Evaluation for Agents 6 lessons

Offline Evaluation for Agents Collecting Evaluation Data Running the Agent on Scenarios Evaluating AI Responses Judge and Alignment Measuring Agent Quality with the Judge

Synthetic Data Generation for Evaluations 4 lessons

Synthetic Evaluation Data Generating Synthetic User Questions Running the Agent on Synthetic Data Running the Judge on Synthetic Data

Week 6 Homework: Evaluating a Recipe Assistant Due Nov 3, 2026 00:59 Europe/Berlin Homework Week 6 Capstone: Evaluating Your Agent Due Nov 3, 2026 00:59 Europe/Berlin Homework Optional Content Optional

Project · weeks 7–9

Capstone Project Nov 2–22 2 topics · 11 lessons Next due Nov 16, 2026 23:59 Europe/Berlin · 4 deadlines

Session 7

Capstone Project Next due Nov 16, 2026 23:59 Europe/Berlin · 4 deadlines 6 lessons

Project Work Project Guidelines Evaluation Criteria Documentation Tips Submission and Peer Review Hackathon

Project attempts

Capstone Project Attempt 1 Submit by Nov 16, 2026 23:59 Europe/Berlin

Capstone Project Attempt 2 Submit by Dec 7, 2026 23:59 Europe/Berlin

Project Attempts

Session 8 Capstone Presentations Optional Content Optional Previous Project Bonus Overview Optional

Optional

Optional 25 topics · 129 lessons

Week 1

Week 1: OpenAI Alternatives 9 lessons

Alternative Providers Overview OpenAI Chat Completions API Groq Anthropic Amazon Bedrock Google Gemini Z.ai Grok Ollama

Week 1: Search Materials 3 lessons

Advanced Search Overview Vector Search Advanced Chunking

Week 2

Week 2: FAQ Assistant 5 lessons

FAQ Assistant Overview Dataset Q&A RAG Use Case: DataTalks.Club FAQ Chatbot Use Case: RAG for Content Management

Week 2: YouTube Transcripts 5 lessons

YouTube Transcripts Getting Transcripts from YouTube Videos Prompt Engineering for Video Summaries Structured Output for Video Summaries Search and RAG with Transcripts

Week 2: PDF Data Extraction (Math Book) 9 lessons

Extracting Information from PDF Books Extracting Text from PDFs Structured Output Cleaning and Refactoring Processing All Book Pages Parallel Processing for Book Pages RAG on a Math Book External Persistent Knowledge Bases Persistent RAG with SQLiteSearch

Week 2: Text Search with Elasticsearch 4 lessons

Elasticsearch Overview Docker Setup for Elasticsearch Ingesting Documents into Elasticsearch Querying Elasticsearch

Week 2: Vector Search with Qdrant 4 lessons

Qdrant Overview Docker Setup for Qdrant Ingesting Documents into Qdrant Querying Qdrant with Vector Search

Week 3

Week 3: Web and YouTube Researcher 6 lessons

Web Research Overview Web Search with OpenAI Built-in Tool Web Search Agent with PydanticAI Web Search with Brave API Web Page Content Extraction YouTube Search and Web Research Agent

Week 3: Tool Calling with Alternative Providers 6 lessons

Tool Calling with Alternative Providers Preparing the Agents Library Tool Calling with OpenAI Responses API Tool Calling with Chat Completions API Tool Calling with Anthropic Tool Calling with Gemini

Week 3: Agent Frameworks - Alternatives 4 lessons

Agent Framework Alternatives OpenAI Agents SDK LangChain Google ADK

Week 3: Model Context Protocol (MCP) 4 lessons

Model Context Protocol Overview Stdio Transport Sse Transport Cursor Integration

Week 4

Week 4: Libraries for Agent Testing 2 lessons

LangWatch Scenario for Agent Testing Evaluation with Evidently

Week 4: Additional Frameworks and Graphs 8 lessons

CrewAI Multi-Agent LangChain Multi-Agent Google ADK Multi-Agent Agentic Graphs Pydantic Graph: Retry Loop Pydantic Graph: Routing Pydantic Graph: Human in the Loop LangGraph

Week 4: Multi-Agent Systems 18 lessons

Multi-Agent Patterns Workflows and Orchestrated Patterns Group 1: Routing Patterns Simple Routing Agents as Tools Handoffs Group 2: Pipeline Patterns Prompt Chaining Static Plan-and-Execute Dynamic Plan-and-Execute Group 3: Feedback Patterns Human-in-the-Loop Fixed Evaluator-Optimizer Dynamic Evaluator-Optimizer Group 4: Coordination Patterns Parallelization Orchestrator-Workers Multi-Agent Collaboration

Week 5

Week 5: Pydantic Logfire and Jaeger 1 lesson

Using Jaeger as OTel Backend

Week 5: DIY Logging and Monitoring 8 lessons

DIY Monitoring Defining the Log Records Integrating the Monitoring Service into the Application Saving Logs and Events to SQLite Creating a Streamlit Dashboard Fake Logs Generator Online Judge Building a Dashboard with Grafana and Postgres

Week 5: Other Observability Platforms 2 lessons

Langwatch Evidently (previous lesson)

Week 6

Week 6: Code Improvements 2 lessons

Refactoring the Evaluation Code Using AI Assistant to Improve the Judge

Week 7

Week 7: My Daily Tasks: End-to-End Project Example 7 lessons

Environment Preparation Implementing Tools from OpenAPI Specs Implementing the Agent with Pydantic AI Organizing the Code into a Project and Creating Streamlit App Testing the Agent Test Judge Monitoring with Pydantic Logfire

Week 7: Guardrails 1 lesson

Guardrails workshop

Week 7: Deployment 4 lessons

Deploying Your Agent Deploying to Streamlit Community Cloud Deploying to Render Deploying to AWS

Week 8

Week 8: Build a Coding Agent from Scratch 6 lessons

Intro Django Template Agent Tools Coding Agent Zai Multi Agent

Week 8: Code Analysis Agent 6 lessons

Introduction Project Scorer Tools Project Scorer Agent File Tools for Code Analysis The Code Analysis Agent Streamlit Demo

Week 8: Deep Research Agent 3 lessons

Deep Research Multi Agent Research Orchestration

Week 8: Student Project Examples 2 lessons

Cohort 1 Previous Project Cases Overview

External course

Enroll via Maven to access this course

This is an external course, run on another platform — it isn't included in any AI Shipping Labs membership plan.

Enroll now