))
Everything you need to know about LLM benchmarks- Turing Test, OpenAI's Healthbench, ARC prize, LM arena
Manage episode 501751128 series 3585389
Dev and Doc에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Dev and Doc 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.
Whenever there was AI, there were benchmarks- from the turing test, to society-changing benchmarks like MNIST and ImageNet to modern problems like the ARC prize, benchmarked served a vital purpose to measure the performance of AI models. But something has shifted in modern times, in the LLM era have benchmarks lost their utility, becoming mere advertisement for big tech?
Even seemingly more sophisticated benchmarks like LM Arena can be gamed by tech giants. We also deep dive into healthcare benchmarks like OpenAI's Healthbench (deeply problematic) and Microsoft's AI-DXO orchestrator agent for diagnosis. Where is this all going? How do we make the perfect benchmark? Or is the real work to be done afterwards in the real world?
👋 Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :)
---
Timestamps
00:00 Intro - The OG benchmarks - Turing test, MNIST, ImageNET
06:40 Are large language models benchmarks similar to humans taking tests?
10:05 Are we testing model capability vs production ready?
12:00 LLM era - data contamination
15:30 LM Arena - The leaderboard illusion paper - how big tech games benchmarks
28:35 Goodhart's law - When a measure becomes a target, it ceases to be a good measure
32:05 Some good benchmarks - games - Pokemon, ARC prize, Minecraft
34:35 Medical benchmarks - OpenAI's healthbench has some big problems
46:50 Microsoft AI-DXO orchestrator for case reports
---
Connect with Us
Your Hosts:
👨🏻⚕️ Doc - Dr. Joshua Au Yeung - LinkedIn
🤖 Dev - Zeljko Kraljevic - Twitter
Follow & Subscribe:
YT: https://youtube.com/@DevAndDoc
Spotify: Follow us on Spotify
Apple Podcasts: Listen on Apple Podcasts
Substack: https://aiforhealthcare.substack.com/
For enquiries:
📧 [email protected]
---
Production Credits
🎞️ Editor: Dragan Kraljević - Instagram
🎨 Brand & Art: Ana Grigorovici - Behance
30 에피소드
공유
