Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Data on Kubernetes Community

Data on Kubernetes Community에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Data on Kubernetes Community 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.

1+ y ago 23:17

MP3•에피소드 홈

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera
Presented by Matt Menzenski, Senior Software Engineering Manager, Payitgov

At PayIt, we’ve been deploying applications to Kubernetes almost since the beginning of the company. Our data workloads, however, have run instead in AWS Glue. This has worked well enough for the reporting use cases that have been the main focus of this team historically. However, at the beginning of 2022, the PayIt data team began building out a new data platform, and in the process, ran into a number of challenges with Glue. In this talk, I will share the difficulties that we encountered with building, deploying, and orchestrating ETL pipelines in AWS Glue, our decision process for moving those workloads into Kubernetes, and the ELT architecture that we’ve arrived at today. Related Links DoKC Website - https://dok.community/ DoKC Meetups - https://www.meetup.com/data-on-kubernetes-community/ Join Slack - https://join.slack.com/t/dokcommunity/shared_invite/zt-1vgv7ymz7-YtLFvZicrcLP9fS3o_r2_w

243 에피소드

#Tech #Demetrios