Inference Engineering Book Club - Week 5: Modalities

We met on 14 September 2026 to discuss chapter 6 of Inference Engineering by Philip Kiely. We used the chapter to look beyond text-to-text systems and discuss what changes when a model serves images, video, speech, or another modality.

Lessons from the session

The discussion left us with these lessons.

• Vision models turn an image into visual tokens through a visual encoder. The group also compared image patches with approaches that work closer to individual pixels.
• Video adds a time dimension to the image's height and width. The discussion used this three-dimensional view to explain why video generation can account for a sequence of frames instead of treating every frame as an unrelated image.
• Serving speech and other non-text-to-text models brings different quality and infrastructure questions. A participant described fine-tuning Whisper for Urdu, Pashto, and Punjabi, where the team had to create training data and still had separate diarization problems after improving transcription.
• The group connected the chapter to local model use. Quantized models can fit smaller hardware, but their output quality may differ from a larger hosted model.
• We also discussed combining handwritten notes, meeting transcripts, and member contributions into a shared book-club resource. You can add notes on the book page or use the member API with an agent.

Next actions

Use this checklist before the final discussion.

• [ ] Read the final chapter of the book before the next meeting.
• [ ] Pick one modality you work with and write down its input representation, latency target, quality measure, and serving constraint. For speech, keep transcription quality and speaker diarization as separate checks.
• [ ] If you have a speech model, collect a small evaluation set that represents your language and speakers. Record transcription errors separately from speaker-attribution errors.
• [ ] Add one useful note, diagram, or question to the Inference Engineering book page so the next reader has something they can use.
• [ ] Bring two book suggestions to the final discussion so the group can choose what to read next.

Recording

Watch the Week 5 recording on YouTube.

Recording chapters

Use these timestamps to jump to the main parts of the discussion.

• 0:03 - Chapter 6 and the modalities overview
• 2:04 - Vision models and visual tokens
• 2:46 - Video generation and temporal context
• 6:02 - Sharing notes and building a common resource
• 9:46 - Serving models beyond text-to-text
• 10:36 - Whisper and low-resource languages
• 14:14 - Fine-tuning data and annotation
• 18:52 - The final chapter and future book choices

Continue with Week 6

The next session covers the final chapter. See the Week 6 event or browse the Inference Engineering Book Club series.