Episode 57: AI Agents and LLM Judges at Scale: Processing Millions of Documents (Without Breaking the Bank)

Vanishing Gradients

Hugo Bowne-Anderson에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Hugo Bowne-Anderson 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.

2M ago 41:27

MP3•에피소드 홈

While many people talk about “agents,” Shreya Shankar (UC Berkeley) has been building the systems that make them reliable. In this episode, she shares how AI agents and LLM judges can be used to process millions of documents accurately and cheaply.

Drawing from work on projects ranging from databases of police misconduct reports to large-scale customer transcripts, Shreya explains the frameworks, error analysis, and guardrails needed to turn flaky LLM outputs into trustworthy pipelines.

We talk through:

Treating LLM workflows as ETL pipelines for unstructured text
Error analysis: why you need humans reviewing the first 50–100 traces
Guardrails like retries, validators, and “gleaning”
How LLM judges work — rubrics, pairwise comparisons, and cost trade-offs
Cheap vs. expensive models: when to swap for savings
Where agents fit in (and where they don’t)

If you’ve ever wondered how to move beyond unreliable demos, this episode shows how to scale LLMs to millions of documents — without breaking the bank.

LINKS

🎓 Learn more:

Hugo's course: Building LLM Applications for Data Scientists and Software Engineers — https://maven.com/s/course/d56067f338

61 에피소드

#Tech #Hugo BowneAnderson Its #Data Science #Machine Learning