Player FM 앱으로 오프라인으로 전환하세요!
Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models
Manage episode 431457150 series 3586723
As head of Product Management for Generative AI at Meta, Joe Spisak leads the team behind Llama, which just released the new 3.1 405B model. We spoke with Joe just two days after the model’s release to ask what’s new, what it enables, and how Meta sees the role of open source in the AI ecosystem.
Joe shares that where Llama 3.1 405B really focused is on pushing scale (it was trained on 15 trillion tokens using 16,000 GPUs) and he’s excited about the zero-shot tool use it will enable, as well as its role in distillation and generating synthetic data to teach smaller models. He tells us why he thinks even frontier models will ultimately commoditize—and why that’s a good thing for the startup ecosystem.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
Open Source AI Is the Way Forward: Mark Zuckerberg essay released with Llama 3.1.
The Bitter Lesson by Rich Sutton
00:00 Introduction
01:28 The Llama 3.1 405B launch
05:02 The open source license
07:01 What's in it for Meta?
10:19 Why not open source?
11:16 Will frontier models commoditize?
12:41 What about startups?
16:29 The Mistral team
19:36 Are all frontier strategies comparable?
22:38 Is model development becoming more like software development?
26:34 Agentic reasoning
29:09 What future levers will unlock reasoning?
31:20 Will coding and math lead to unlocks?
33:09 Small models
34:08 7X more data
37:36 Are we going to hit a wall?
39:49 Lightning round
19 에피소드
Manage episode 431457150 series 3586723
As head of Product Management for Generative AI at Meta, Joe Spisak leads the team behind Llama, which just released the new 3.1 405B model. We spoke with Joe just two days after the model’s release to ask what’s new, what it enables, and how Meta sees the role of open source in the AI ecosystem.
Joe shares that where Llama 3.1 405B really focused is on pushing scale (it was trained on 15 trillion tokens using 16,000 GPUs) and he’s excited about the zero-shot tool use it will enable, as well as its role in distillation and generating synthetic data to teach smaller models. He tells us why he thinks even frontier models will ultimately commoditize—and why that’s a good thing for the startup ecosystem.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
Open Source AI Is the Way Forward: Mark Zuckerberg essay released with Llama 3.1.
The Bitter Lesson by Rich Sutton
00:00 Introduction
01:28 The Llama 3.1 405B launch
05:02 The open source license
07:01 What's in it for Meta?
10:19 Why not open source?
11:16 Will frontier models commoditize?
12:41 What about startups?
16:29 The Mistral team
19:36 Are all frontier strategies comparable?
22:38 Is model development becoming more like software development?
26:34 Agentic reasoning
29:09 What future levers will unlock reasoning?
31:20 Will coding and math lead to unlocks?
33:09 Small models
34:08 7X more data
37:36 Are we going to hit a wall?
39:49 Lightning round
19 에피소드
Alla avsnitt
×플레이어 FM에 오신것을 환영합니다!
플레이어 FM은 웹에서 고품질 팟캐스트를 검색하여 지금 바로 즐길 수 있도록 합니다. 최고의 팟캐스트 앱이며 Android, iPhone 및 웹에서도 작동합니다. 장치 간 구독 동기화를 위해 가입하세요.