Multimodal Evals:🦄 #34

Rather than hand-labeling receipts with correct answers, Kevin designed evals based on mathematical invariants - does the subtotal plus tax equal the grand total? Do unit prices times quantities equal line totals? (7:45) This approach lets you evaluate extraction quality without any ground truth labels, which mirrors real-world conditions where golden data sets rarely exist.

The very first step before writing any code was simply looking at the receipt images to understand their characteristics (18:30) - discovering they were Indonesian currency, had varying sizes, grease stains, shadows, crinkles, and unusual features like restaurant-specific taxes (PB1). This manual data exploration directly informed which fields needed to be in the data model.

The extraction schema evolved through multiple iterations (24:00). The initial model only had item name, quantity, unit price, total price, and a simple total. Through eval failures, Kevin discovered he needed subtotal, tax, rounding, discounts, and unit discounts - each added after observing specific failure patterns in the dashboard.

One of the biggest improvements came simply from switching models (29:30). Moving from GPT-4o to Gemini 2.5 Flash with the exact same prompt and data model produced dramatically better results, suggesting Gemini Flash has notably better multimodal OCR capabilities than competitors for this type of structured extraction.