Solving a Real AI Engineer Take-Home Assignment Live

We solve a real AI engineer take-home assignment live. The product is a small investment coaching bot that answers public-company research questions without crossing into personalized financial advice.

The flow of the session:

• Brainstorm scope and survey data sources.
• Build a CLI agent with PydanticAI and OpenAI.
• Switch from a paid market-data API to free SEC EDGAR data after the free tier blocks us.
• Clean up the pointless tests the coding agent generated and write test guidelines we actually want to follow.
• Add a small evaluation harness with scenarios and an LLM judge.
• Wrap the result as a Telegram bot.

We do every step with a coding agent. We include the prompts verbatim, so you can repeat the same steps with any agent you like.

Links

Useful resources for this workshop:

• AI engineering take-home assignments research
• AI engineering field guide: home assignments

The app you will build

The final app looks like this:

The agent has four tools, all backed by free SEC EDGAR endpoints:

• search_company(query) resolves a ticker or company name to a CIK.
• get_financial_snapshot(ticker_or_cik) fetches recent annual revenue, profit, balance sheet, and cash facts from companyfacts XBRL data.
• get_latest_filings(ticker_or_cik) returns recent filing metadata and SEC URLs.
• get_filing_digest(ticker_or_cik, form_type) pulls the latest 10-K or 10-Q text and extracts business, revenue, risk, and MD&A snippets.

The agent's instructions set a strict safety boundary:

• no buy/sell/hold recommendations
• no position sizing
• no predictions of price direction

The agent turns a buy/sell question into an educational research brief instead.

The evaluation harness is in evals/manual/. It runs a list of scenarios from scenarios.csv through the agent and records the full trajectory (the sequence of tool calls the agent made). An LLM-as-judge then scores whether the answer stayed inside the safety boundary and answered the question.