The DuckLake Lakehouse Format // Hannes Mühleisen // #339

MLOps.community

The DuckLake Lakehouse Format // MLOps Podcast #339 with Hannes Mühleisen, Co-founder and CEO of DuckDB Labs.

Join the Community: https://go.mlops.community/YTJoinIn

Get the newsletter: https://go.mlops.community/YTNewsletter

// Abstract

Managing data on Object Stores has been a painful affair. Users had to choose between data swamp chaos or a maze of metadata files with catalog servers on top.

DuckLake is a new paradigm for managing data on object stores: First, it uses classical SQL data management systems to manage metadata. Second, actual data is stored in Parquet files on pretty arbitrary storage. Third, processing queries is done client-side, or anywhere really. DuckDB is the first system to integrate with DuckLake using an extension with the same name.

Conceptually, DuckLake enables central control over truth while decentralizing compute and storage entirely. DuckLake turns data warehouse architecture upside down by departing from the integrated metadata/compute layer towards a fully disconnected operation with only centralized metadata. For the first time, DuckLake allows a “multi-player” experience with DuckDB, where computation stays fully local, but transactional control is centralized.

// Bio

Hannes Mühleisen 🔈 is a creator of the DuckDB database management system and Co-founder and CEO of DuckDB Labs. He is a senior researcher at the Centrum Wiskunde & Informatica (CWI) in Amsterdam. He is also Professor of Data Engineering at Radboud University Nijmegen.

// Related Links

Website: https://hannes.muehleisen.orgUnleashing Unconstrained News Knowledge Graphs to Combat Misinformation // Robert Caulk // #279 - https://youtu.be/pF8zTI867EI

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

Join our Slack community [https://go.mlops.community/slack]

Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)]

Sign up for the next meetup: [https://go.mlops.community/register]

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Hudson on LinkedIn: /hfmuehleisen

Timestamps:

[00:00] Spooky ease in tech

[00:29] DuckDB and DuckLake

[07:50] Pain vs trust factors

[13:12] Prioritizing project features

[16:16] Platform growth tension

[22:06] Building principles

[25:26] OSS vs system reliability

[30:27] Creative uses of DuckDB

[35:35] Tecton product strategy

[43:30] Mindset shift

[52:25] DuckDB future shifts

[55:37] Wrap up