Artwork

เนื้อหาจัดทำโดย Data on Kubernetes Community เนื้อหาพอดแคสต์ทั้งหมด รวมถึงตอน กราฟิก และคำอธิบายพอดแคสต์ได้รับการอัปโหลดและจัดเตรียมโดย Data on Kubernetes Community หรือพันธมิตรแพลตฟอร์มพอดแคสต์โดยตรง หากคุณเชื่อว่ามีบุคคลอื่นใช้งานที่มีลิขสิทธิ์ของคุณโดยไม่ได้รับอนุญาต คุณสามารถปฏิบัติตามขั้นตอนที่อธิบายไว้ที่นี่ https://th.player.fm/legal
Player FM - แอป Podcast
ออฟไลน์ด้วยแอป Player FM !

DoK Talks #99- ETL/ELT on Kubernetes with Airbyte: K8s Development Insights // Abhi Vaidyanatha

1:00:45
 
แบ่งปัน
 

Manage episode 306403268 series 2865115
เนื้อหาจัดทำโดย Data on Kubernetes Community เนื้อหาพอดแคสต์ทั้งหมด รวมถึงตอน กราฟิก และคำอธิบายพอดแคสต์ได้รับการอัปโหลดและจัดเตรียมโดย Data on Kubernetes Community หรือพันธมิตรแพลตฟอร์มพอดแคสต์โดยตรง หากคุณเชื่อว่ามีบุคคลอื่นใช้งานที่มีลิขสิทธิ์ของคุณโดยไม่ได้รับอนุญาต คุณสามารถปฏิบัติตามขั้นตอนที่อธิบายไว้ที่นี่ https://th.player.fm/legal

https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
ETL/ELT on Kubernetes is currently an unsolved problem. There are a lot of different approaches vying for a spot as the de facto method, but none are clear winners. Considering that the cloud-native landscape is built for deploying Dockerized, open-source software, many of the closed-source solutions fall flat and don't mesh with the trajectory of the community.
Airbyte is an open-source ETL/ELT tool that harmonizes well with the cloud-native landscape and lives to enable your stateful workloads on Kubernetes. Previously, I have talked about a theoretical deployment on Kubernetes and the nuances behind deploying an ETL/ELT pipeline in such an environment. Now, I'm looking to follow that up with how we actually implemented that strategy as we launched our K8s beta. Additionally, I'll dive into some of the nitty gritty details that we needed to figure out in order to get this all working... stuff that isn't really found online!
Overall, this will be a really unique case of getting to do a retrospective on what we planned our architecture to look like and following up with some great development insights as we solidified the final implementation.
KEY TAKE-AWAYS FROM THE TALK
- Quick overview of Airbyte and open-source ETL/ELT [5 minutes]
- Why run your ETL/ELT in K8s? [3 minutes]
- A quick recap on the previous talk (what we thought the architecture would look like) [5 minutes]
- Display the actual architecture and implementation [10 minutes]
-> Talk about how to communicate with k8s pods on STDOUT and STDIN pipes
-> Describe parent-child process termination strategy
-> Describe persistence layer/strategy and config storage
- Quick demo of an Airbyte deployment on K8s [10 minutes]
BIO
Abhi is a confused economist who enjoys writing backend code for data management software. He now spends most of his time doing developer relations in the data integration space, where he looks to evangelize open source technologies. In his spare time he is a DJ, drummer, and competitive Super Smash Bros. player. He is a staunch advocate of proper semicolon usage, Oxford commas and overused grammar jokes.

  continue reading

243 ตอน

Artwork
iconแบ่งปัน
 
Manage episode 306403268 series 2865115
เนื้อหาจัดทำโดย Data on Kubernetes Community เนื้อหาพอดแคสต์ทั้งหมด รวมถึงตอน กราฟิก และคำอธิบายพอดแคสต์ได้รับการอัปโหลดและจัดเตรียมโดย Data on Kubernetes Community หรือพันธมิตรแพลตฟอร์มพอดแคสต์โดยตรง หากคุณเชื่อว่ามีบุคคลอื่นใช้งานที่มีลิขสิทธิ์ของคุณโดยไม่ได้รับอนุญาต คุณสามารถปฏิบัติตามขั้นตอนที่อธิบายไว้ที่นี่ https://th.player.fm/legal

https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
ETL/ELT on Kubernetes is currently an unsolved problem. There are a lot of different approaches vying for a spot as the de facto method, but none are clear winners. Considering that the cloud-native landscape is built for deploying Dockerized, open-source software, many of the closed-source solutions fall flat and don't mesh with the trajectory of the community.
Airbyte is an open-source ETL/ELT tool that harmonizes well with the cloud-native landscape and lives to enable your stateful workloads on Kubernetes. Previously, I have talked about a theoretical deployment on Kubernetes and the nuances behind deploying an ETL/ELT pipeline in such an environment. Now, I'm looking to follow that up with how we actually implemented that strategy as we launched our K8s beta. Additionally, I'll dive into some of the nitty gritty details that we needed to figure out in order to get this all working... stuff that isn't really found online!
Overall, this will be a really unique case of getting to do a retrospective on what we planned our architecture to look like and following up with some great development insights as we solidified the final implementation.
KEY TAKE-AWAYS FROM THE TALK
- Quick overview of Airbyte and open-source ETL/ELT [5 minutes]
- Why run your ETL/ELT in K8s? [3 minutes]
- A quick recap on the previous talk (what we thought the architecture would look like) [5 minutes]
- Display the actual architecture and implementation [10 minutes]
-> Talk about how to communicate with k8s pods on STDOUT and STDIN pipes
-> Describe parent-child process termination strategy
-> Describe persistence layer/strategy and config storage
- Quick demo of an Airbyte deployment on K8s [10 minutes]
BIO
Abhi is a confused economist who enjoys writing backend code for data management software. He now spends most of his time doing developer relations in the data integration space, where he looks to evangelize open source technologies. In his spare time he is a DJ, drummer, and competitive Super Smash Bros. player. He is a staunch advocate of proper semicolon usage, Oxford commas and overused grammar jokes.

  continue reading

243 ตอน

ทุกตอน

×
 
Loading …

ขอต้อนรับสู่ Player FM!

Player FM กำลังหาเว็บ

 

คู่มืออ้างอิงด่วน