BI 120 James Fitzgerald, Andrew Saxe, Weinan Sun: Optimizing Memories

Brain Inspired

Support the show to get full episodes, full archive, and join the Discord community.

James, Andrew, and Weinan discuss their recent theory about how the brain might use complementary learning systems to optimize our memories. The idea is that our hippocampus creates our episodic memories for individual events, full of particular details. And through a complementary process, slowly consolidates those memories within our neocortex through mechanisms like hippocampal replay. The new idea in their work suggests a way for the consolidated cortical memory to become optimized for generalization, something humans are known to be capable of but deep learning has yet to build. We discuss what their theory predicts about how the "correct" process depends on how much noise and variability there is in the learning environment, how their model solves this, and how it relates to our brain and behavior.

• James' Janelia page.
• Weinan's Janelia page.
• Andrew's website.
• Twitter:
  • Andrew: @SaxeLab
  • Weinan: @sunw37
• Paper we discuss:
  • Organizing memories for generalization in complementary learning systems.
• Andrew's previous episode: BI 052 Andrew Saxe: Deep Learning Theory

0:00 - Intro 3:57 - Guest Intros 15:04 - Organizing memories for generalization 26:48 - Teacher, student, and notebook models 30:51 - Shallow linear networks 33:17 - How to optimize generalization 47:05 - Replay as a generalization regulator 54:57 - Whole greater than sum of its parts 1:05:37 - Unpredictability 1:10:41 - Heuristics 1:13:52 - Theoretical neuroscience for AI 1:29:42 - Current personal thinking