Artwork

Felipe Flores에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Felipe Flores 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.
Player FM -팟 캐스트 앱
Player FM 앱으로 오프라인으로 전환하세요!

#240: Overcoming the challenges facing modern data engineering teams

43:13
 
공유
 

Manage episode 371105548 series 2310475
Felipe Flores에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Felipe Flores 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.

This week on the Data Futurology podcast we host Paul Milinkovic, the APAC Regional Director for the leading data integration platform, StreamSets. Milinkovic joins us to share his insights into data engineers' challenges and the pipelines they manage and maintain.

One statistic really highlights just how challenging work environments have become for data engineers: 76 per cent of organisations have a pipeline break at least monthly and for 36 per cent, it's weekly. Rather than contributing strategically to their organisations, engineers split their time between diagnosis and repair, and building new pipelines. This costs the organisation, as half the time the engineer isn’t being used strategically. It also leads to cultures of over-working, burnout, and high levels of churn within the data engineering team.

Another challenge data teams struggle with is competing priorities. When multiple lines of business need pipelines developed, teams often need to triage to accommodate priority tasks, and this affects overall company outcomes. Being able to help organisations deliver a low or no-code environment that is highly visual and accessible to non-data specialists has been a critical benefit for organisations that have adopted StreamSets.

Milinkovic then shares two case studies where StreamSets has helped with overcoming these challenges. In one, a bank achieved a seemingly impossible task – becoming compliant with looming Consumer Data Act requirements within four months. Then, a second bank was able to leverage StreamSets to its data to detect and thwart $9 million in fraudulent activity in a single month.

For more deep insights into overcoming the challenges facing modern data engineering teams, tune into the podcast!

Links

Website: https://streamsets.com

Follow on LinkedIn: https://www.linkedin.com/company/streamsets/

Whitepapers:

https://go.streamsets.com/Whitepaper-Dollars_and_Sense_UGLP.html?utm_medium=website&utm_source=DataFuturology&utm_campaign=eg_dollars_and_sense_of_dataops

https://go.streamsets.com/Whitepaper-Dollars_and_Sense_UGLP.html?utm_medium=website&utm_source=DataFuturology&utm_campaign=eg_dollars_and_sense_of_dataops

https://go.streamsets.com/230214-lifting-the-lid-on-data-integration-UGLP.html?utm_me[…]turology&utm_campaign=eg_lifting_the_lid_on_data_integration

What we discussed:

00:00 Introduction

02:22: Felipe introduces Paul Milinkovic.

03:38: Milinkovic shares his background and his history with data at various levels and applications.

06:04: Milinkovic overviews StreamSets – when and why the company was founded, and what its core capabilities are.

09:04: What are the main issues that StreamSets helps data engineering teams solve?

12:57: How does StreamSets address traditional data pipeline design and build challenges?

12:33: What are the benefits of having a solution that is visual and accessible to non-technical users?

22:51: One of the common questions with the self-service approach to data is governance. How can that be handled while still allowing full flexibility?

26:46: Data engineers care a great deal about the quality and accuracy of data and the platforms that it sits on. Milinkovic explains why it is so important that they have the tools to be able to deliver that to the organisation.

31:24: What is the financial impact of data engineering teams spending as much time fixing pipelines as they are?

33:49: Milinkovic shares some case studies and use cases to highlight the value of StreamSets’ approach to data engineering.

  continue reading

268 에피소드

Artwork
icon공유
 
Manage episode 371105548 series 2310475
Felipe Flores에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Felipe Flores 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.

This week on the Data Futurology podcast we host Paul Milinkovic, the APAC Regional Director for the leading data integration platform, StreamSets. Milinkovic joins us to share his insights into data engineers' challenges and the pipelines they manage and maintain.

One statistic really highlights just how challenging work environments have become for data engineers: 76 per cent of organisations have a pipeline break at least monthly and for 36 per cent, it's weekly. Rather than contributing strategically to their organisations, engineers split their time between diagnosis and repair, and building new pipelines. This costs the organisation, as half the time the engineer isn’t being used strategically. It also leads to cultures of over-working, burnout, and high levels of churn within the data engineering team.

Another challenge data teams struggle with is competing priorities. When multiple lines of business need pipelines developed, teams often need to triage to accommodate priority tasks, and this affects overall company outcomes. Being able to help organisations deliver a low or no-code environment that is highly visual and accessible to non-data specialists has been a critical benefit for organisations that have adopted StreamSets.

Milinkovic then shares two case studies where StreamSets has helped with overcoming these challenges. In one, a bank achieved a seemingly impossible task – becoming compliant with looming Consumer Data Act requirements within four months. Then, a second bank was able to leverage StreamSets to its data to detect and thwart $9 million in fraudulent activity in a single month.

For more deep insights into overcoming the challenges facing modern data engineering teams, tune into the podcast!

Links

Website: https://streamsets.com

Follow on LinkedIn: https://www.linkedin.com/company/streamsets/

Whitepapers:

https://go.streamsets.com/Whitepaper-Dollars_and_Sense_UGLP.html?utm_medium=website&utm_source=DataFuturology&utm_campaign=eg_dollars_and_sense_of_dataops

https://go.streamsets.com/Whitepaper-Dollars_and_Sense_UGLP.html?utm_medium=website&utm_source=DataFuturology&utm_campaign=eg_dollars_and_sense_of_dataops

https://go.streamsets.com/230214-lifting-the-lid-on-data-integration-UGLP.html?utm_me[…]turology&utm_campaign=eg_lifting_the_lid_on_data_integration

What we discussed:

00:00 Introduction

02:22: Felipe introduces Paul Milinkovic.

03:38: Milinkovic shares his background and his history with data at various levels and applications.

06:04: Milinkovic overviews StreamSets – when and why the company was founded, and what its core capabilities are.

09:04: What are the main issues that StreamSets helps data engineering teams solve?

12:57: How does StreamSets address traditional data pipeline design and build challenges?

12:33: What are the benefits of having a solution that is visual and accessible to non-technical users?

22:51: One of the common questions with the self-service approach to data is governance. How can that be handled while still allowing full flexibility?

26:46: Data engineers care a great deal about the quality and accuracy of data and the platforms that it sits on. Milinkovic explains why it is so important that they have the tools to be able to deliver that to the organisation.

31:24: What is the financial impact of data engineering teams spending as much time fixing pipelines as they are?

33:49: Milinkovic shares some case studies and use cases to highlight the value of StreamSets’ approach to data engineering.

  continue reading

268 에피소드

모든 에피소드

×
 
Loading …

플레이어 FM에 오신것을 환영합니다!

플레이어 FM은 웹에서 고품질 팟캐스트를 검색하여 지금 바로 즐길 수 있도록 합니다. 최고의 팟캐스트 앱이며 Android, iPhone 및 웹에서도 작동합니다. 장치 간 구독 동기화를 위해 가입하세요.

 

빠른 참조 가이드