Advanced Lakehouse Management With The LakeKeeper Iceberg REST Catalog

Data Engineering Podcast

Tobias Macey에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Tobias Macey 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.

6M ago 57:13

MP3•에피소드 홈

Summary
In this episode of the Data Engineering Podcast Viktor Kessler, co-founder of Vakmo, talks about the architectural patterns in the lake house enabled by a fast and feature-rich Iceberg catalog. Viktor shares his journey from data warehouses to developing the open-source project, Lakekeeper, an Apache Iceberg REST catalog written in Rust that facilitates building lake houses with essential components like storage, compute, and catalog management. He discusses the importance of metadata in making data actionable, the evolution of data catalogs, and the challenges and innovations in the space, including integration with OpenFGA for fine-grained access control and managing data across formats and compute engines.
Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Viktor Kessler about architectural patterns in the lakehouse that are unlocked by a fast and feature-rich Iceberg catalog

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what LakeKeeper is and the story behind it?
- What is the core of the problem that you are addressing?
There has been a lot of activity in the catalog space recently. What are the driving forces that have highlighted the need for a better metadata catalog in the data lake/distributed data ecosystem?
- How would you characterize the feature sets/problem spaces that different entrants are focused on addressing?
Iceberg as a table format has gained a lot of attention and adoption across the data ecosystem. The REST catalog format has opened the door for numerous implementations. What are the opportunities for innovation and improving user experience in that space?
What is the role of the catalog in managing security and governance? (AuthZ, auditing, etc.)
- What are the channels for propagating identity and permissions to compute engines? (how do you avoid head-scratching about permission denied situations)
Can you describe how LakeKeeper is implemented?
- How have the design and goals of the project changed since you first started working on it?
For someone who has an existing set of Iceberg tables and catalog, what does the migration process look like?
What new workflows or capabilities does LakeKeeper enable for data teams using Iceberg tables across one or more compute frameworks?
What are the most interesting, innovative, or unexpected ways that you have seen LakeKeeper used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on LakeKeeper?
When is LakeKeeper the wrong choice?
What do you have planned for the future of LakeKeeper?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

486 에피소드

#Tech #Tobias Macey #Data Engineering #Big Data #Database #Data Pipeline #Podcasting Education