))
Everything you need to know about LLM benchmarks- Turing Test, OpenAI's Healthbench, ARC prize, LM arena
Manage episode 501751128 series 3585389
เนื้อหาจัดทำโดย Dev and Doc เนื้อหาพอดแคสต์ทั้งหมด รวมถึงตอน กราฟิก และคำอธิบายพอดแคสต์ได้รับการอัปโหลดและจัดหาให้โดยตรงจาก Dev and Doc หรือพันธมิตรแพลตฟอร์มพอดแคสต์ของพวกเขา หากคุณเชื่อว่ามีบุคคลอื่นใช้งานที่มีลิขสิทธิ์ของคุณโดยไม่ได้รับอนุญาต คุณสามารถปฏิบัติตามขั้นตอนที่แสดงไว้ที่นี่ https://th.player.fm/legal
Whenever there was AI, there were benchmarks- from the turing test, to society-changing benchmarks like MNIST and ImageNet to modern problems like the ARC prize, benchmarked served a vital purpose to measure the performance of AI models. But something has shifted in modern times, in the LLM era have benchmarks lost their utility, becoming mere advertisement for big tech?
Even seemingly more sophisticated benchmarks like LM Arena can be gamed by tech giants. We also deep dive into healthcare benchmarks like OpenAI's Healthbench (deeply problematic) and Microsoft's AI-DXO orchestrator agent for diagnosis. Where is this all going? How do we make the perfect benchmark? Or is the real work to be done afterwards in the real world?
👋 Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :)
---
Timestamps
00:00 Intro - The OG benchmarks - Turing test, MNIST, ImageNET
06:40 Are large language models benchmarks similar to humans taking tests?
10:05 Are we testing model capability vs production ready?
12:00 LLM era - data contamination
15:30 LM Arena - The leaderboard illusion paper - how big tech games benchmarks
28:35 Goodhart's law - When a measure becomes a target, it ceases to be a good measure
32:05 Some good benchmarks - games - Pokemon, ARC prize, Minecraft
34:35 Medical benchmarks - OpenAI's healthbench has some big problems
46:50 Microsoft AI-DXO orchestrator for case reports
---
Connect with Us
Your Hosts:
👨🏻⚕️ Doc - Dr. Joshua Au Yeung - LinkedIn
🤖 Dev - Zeljko Kraljevic - Twitter
Follow & Subscribe:
YT: https://youtube.com/@DevAndDoc
Spotify: Follow us on Spotify
Apple Podcasts: Listen on Apple Podcasts
Substack: https://aiforhealthcare.substack.com/
For enquiries:
📧 [email protected]
---
Production Credits
🎞️ Editor: Dragan Kraljević - Instagram
🎨 Brand & Art: Ana Grigorovici - Behance
30 ตอน
แบ่งปัน
