Building Food Metadata with LLM Juries, Context Optimization & Multimodal AI

Core View

DoorDash uses a multimodal AI platform to infer item- and store-level attributes from text, images, and web search signals.
An LLM jury system improves annotation accuracy by ~20% compared to human reviewers by enabling consensus-based evaluation of tags.
Context-optimization agents iteratively improve prompts using failure signals, increasing model precision by over 20% and accelerating development tenfold.
Distributed computing reduces backfill time from over a month to just days, enabling real-time metadata updates at scale.
AI-led annotation generates training data with 90% lower inference cost and zero human effort, accelerating fine-tuning of specialized models.

LLM jury system: Multiple LLM evaluators independently judge tags and vote on consensus, improving accuracy and reducing human dependency.
Context optimization loop: Failure signals from high-quality evaluation datasets are used to propose and test prompt improvements, mimicking reinforcement learning.
Deduplication, Spark distribution, batch processing, and result remapping enable efficient, scalable inference across millions of items.
AI-generated training data accelerates fine-tuning without requiring human annotation, reducing development cycle time significantly.

Structured metadata enables powerful downstream features: customer search, personalization, filtering, and analytics.
Enhanced search and discovery allow users to find relevant dishes based on cuisine, dietary needs, and preferences.
Metadata serves as a foundational layer for personalization and analytics across the DoorDash platform.

Multimodal AI systems can achieve high accuracy and scalability when built with automated evaluation and iterative context optimization.
LLM juries and failure-driven prompt tuning reduce human labor and improve model precision in large-scale production environments.
Distributed inference pipelines are essential for handling millions of items with low latency and cost efficiency.

Topics: Tech
Tags: tech multimodal-ai llm