Solving a Real AI Engineer Take-Home Assignment Live
Event times are shown in Europe/Berlin. Sign in to use your timezone
Continue with the workshop writeup
Open the canonical pages, recording, materials, and code repo.
We solve a real AI engineer take-home assignment live. The product is a small investment coaching bot that answers public-company research questions without crossing into personalized financial advice.
The flow of the session:
- Brainstorm scope and survey data sources.
- Build a CLI agent with PydanticAI and OpenAI.
- Switch from a paid market-data API to free SEC EDGAR data after the free tier blocks us.
- Clean up the pointless tests the coding agent generated and write test guidelines we actually want to follow.
- Add a small evaluation harness with scenarios and an LLM judge.
- Wrap the result as a Telegram bot.
We do every step with a coding agent. We include the prompts verbatim, so you can repeat the same steps with any agent you like.
Links
Useful resources for this workshop:
The app you will build
The final app looks like this:
The agent has four tools, all backed by free SEC EDGAR endpoints:
search_company(query)resolves a ticker or company name to a CIK.get_financial_snapshot(ticker_or_cik)fetches recent annual revenue, profit, balance sheet, and cash facts from companyfacts XBRL data.get_latest_filings(ticker_or_cik)returns recent filing metadata and SEC URLs.get_filing_digest(ticker_or_cik, form_type)pulls the latest 10-K or 10-Q text and extracts business, revenue, risk, and MD&A snippets.
The agent's instructions set a strict safety boundary:
- no buy/sell/hold recommendations
- no position sizing
- no predictions of price direction
The agent turns a buy/sell question into an educational research brief instead.
The evaluation harness is in evals/manual/. It runs a list of
scenarios from scenarios.csv through the agent and records the full
trajectory (the sequence of tool calls the agent made). An LLM-as-judge
then scores whether the answer stayed inside the safety boundary and
answered the question.
Hosted by
Alexey Grigorev
Chief Agent Officer at AI Shipping Labs
Software engineer and machine learning practitioner with 15+ years of experience building production ML systems. I focus on practical, production-grade ML and AI systems, from early prototypes to reliable systems in production.
I'm the founder of DataTalks.Club, a free community that connects tens of thousands of practitioners worldwide, and the creator of the Zoomcamp series, free, code-first programs that have reached 100,000+ learners globally.
At AI Shipping Labs, I'm building the kind of environment that would have accelerated my own career growth. After years of teaching at scale, I wanted something more focused: a space for action-oriented builders who want to turn AI ideas into real projects. The community gives members the structure, accountability, and peer support to ship practical AI products consistently, even alongside their main jobs.
More from AI Shipping Labs
AI Dev Tools Office Hours — Session 4
Open office hours to discuss AI dev tools, workflows, and shipping AI products. Bring your questions.
September 29, 2026
Jev for LLM Evaluations
In the batch and flex workshop we built an eval for the FAQ agent and scored it with an LLM judge. Here we swap that judge for Jev, a model that returns typed decisions instead of…
September 22, 2026
AI Engineering Buildcamp: From RAG to Agents
A hands-on course on building production AI applications with large language models: retrieval-augmented generation (RAG), structured output, AI agents, testing, monitoring, and e…