Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

The Data Flowcast에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 The Data Flowcast 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.

6M ago 27:40

MP3•에피소드 홈

Keeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov, Senior Data Engineer at Stripe, and Sharadh Krishnamurthy, Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights on scaling data workflows while maintaining performance, reliability and developer experience.

Key Takeaways:

(02:04) Stripe’s mission is to grow the GDP of the internet by supporting businesses with payments and data.

(05:08) 80% of Stripe engineers use data orchestration, making scalability critical.

(06:06) Airflow powers business reports, regulatory needs and ML workflows.

(08:02) Custom task frameworks improve dependencies and validation.

(08:50) "User scope mode" enables local testing without production impact.

(10:39) Migrating to Airflow 2 improves isolation, safety and scalability.

(16:40) Monolithic DAGs caused database issues, prompting a service-based shift.

(19:24) Frequent Airflow upgrades ensure stability and access to new features.

(21:38) DAG versioning and backfill improvements enhance developer experience.

(23:38) Greater UI customization would offer more flexibility.

Resources Mentioned:

Nick Bilozerov -

https://www.linkedin.com/in/nick-bilozerov/

Sharadh Krishnamurthy -

https://www.linkedin.com/in/sharadhk/

Apache Airflow -

https://airflow.apache.org/

Stripe | LinkedIn -

https://www.linkedin.com/company/stripe/

Stripe | Website -

https://stripe.com/

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

68 에피소드

#Tech #Airflow