พอดคาสท์ Data Engineering ชั้นนำ (2024)

1
Making Email Better With AI At Shortwave 53:43

10h ago53:43

53:43

Summary Generative AI has rapidly transformed everything in the technology sector. When Andrew Lee started work on Shortwave he was focused on making email more productive. When AI started gaining adoption he realized that he had even more potential for a transformative experience. In this episode he shares the technical challenges that he and his …

1
How ZoomInfo transitioned from data graveyards to ROI-driven data projects 39:46

7d ago39:46

39:46

Too often expensive resources and manhours are spent on dashboards no one uses, resulting in zero ROI. Philip Philip Zelitchenko, VP of Data & Analytics at ZoomInfo met the bros to talk about adopting product management principles to ensure data projects have value, and provide an unfiltered peak into ZoomInfo’s data stack and unique tech culture.…

1
Designing A Non-Relational Database Engine 1:16:01

8d ago1:16:01

1:16:01

Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing …

1
Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer 56:23

15d ago56:23

56:23

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. In order to enable this use case, while maintaining a single point of access, the semantic layer has evolved as a technological sol…

1
Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary 50:44

23d ago50:44

50:44

Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. While there are numerous products available to provide that visibility, they all have different technolo…

1
Matthew Weingarten from Disney Streaming about Data Quality Best Practices 27:21

28d ago27:21

27:21

Matthew Weingarten, Lead Data Engineer at Disney Streaming, talks about principles essential for data quality, cost optimization, debugging, and data modeling, as adopted by the world's leading companies.โดย The Firebolt Data Bros

1
Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+ 55:39

29d ago55:39

55:39

Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned commercial companion to the open source project they are investing in that capability with a suite of new features. In this episode Pe…

1
Reconciling The Data In Your Databases With Datafold 58:14

1M ago58:14

58:14

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. In this episode Gleb Mezhanskiy, founder and CEO of Datafold,…

1
Version Your Data Lakehouse Like Your Software With Nessie 40:55

1M ago40:55

40:55

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. The primary purpose of the catalog is to inform the query engine of what data exists and where, but the Nessie project aims to go beyond…

1
Iceberg at Netflix and Beyond with Ryan Blue 47:37

2M ago47:37

47:37

Apache Iceberg is an open source high-performance format for huge data tables. Iceberg enables the use of SQL tables for big data, while making it possible for engines like Spark and Hive to safely work with the same tables, at the same time. Iceberg was started at Netflix by Ryan Blue and Dan Weeks, and was open-sourced and donated to the Apache S…

1
When And How To Conduct An AI Program 46:25

2M ago46:25

46:25

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. In order to make those promises a reality there is a substantial amount of strategy and investment required. Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about h…

1
Joseph Machado, Senior Data Engineer @ LinkedIn talks best practices 25:59

2M ago25:59

25:59

Data engineering should be less about the stack and more about best practices. While tools may change, foundational principles will remain constant. Joseph Mercado, Senior Data Engineer at LinkedIn, is on The Data Engineering Show to talk about principles that are key to success, leveraging AI for automation, and adopting software engineering metho…

1
Find Out About The Technology Behind The Latest PFAD In Analytical Database Development 56:00

2M ago56:00

56:00

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow …

1
Using Trino And Iceberg As The Foundation Of Your Data Lakehouse 58:46

2M ago58:46

58:46

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a reality. In this episode Dain Sundstrom, CTO of Starburst, explains how the combinatio…

1
Data Sharing Across Business And Platform Boundaries 59:55

2M ago59:55

59:55

Summary Sharing data is a simple concept, but complicated to implement well. There are numerous business rules and regulatory concerns that need to be applied. There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. In this episode Andrew Jefferson explains …

1
Building a Data Lake with Adam Ferrari 46:19

3M ago46:19

46:19

Starburst is a data lake analytics platform. It’s designed to help users work with structured data at scale, and is built on the open source platform, Trino. Adam Ferrari is the SVP of Engineering at Starburst. He joins the show to talk about Starburst, data engineering, and what it takes to build a data lake. Full Disclosure: Starburst is a sponso…

1
Tackling Real Time Streaming Data With SQL Using RisingWave 56:55

3M ago56:55

56:55

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continu…

1
Build A Data Lake For Your Security Logs With Scanner 1:02:38

3M ago1:02:38

1:02:38

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much effort to structure the logs, or aren't fast enough for interactive use cases. Cliff Crosland co-founded Scanner to provide fast querying…

1
Professors Joe Hellerstein and Joseph Gonzalez on LLMs 46:07

3M ago46:07

46:07

Joe Hellerstein is the Jim Gray Professor of Computer Science at Berkeley and Joseph Gonzalez is an Associate Professor in the Electrical Engineering and Computer Science department. They’ve inspired generations of database enthusiasts (including Benji and Eldad) and have come on the show to talk about all things LLM and RunLLM which they co-founde…

1
Modern Customer Data Platform Principles 1:01:33

3M ago1:01:33

1:01:33

Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how…

1
Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel 50:26

4M ago50:26

50:26

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. As the sophistication increases, so does the complexity, leading to challenges for user ex…

1
Megan Lieu on powerful notebooks that enable collaboration 31:31

4M ago31:31

31:31

There are two types of data influencers on LinkedIn: 1. Those who talk directly about the products and companies they work for 2. Those that provide more general guidance, tips and opinions Can influencers actually be passionate about the products they’re developing and straightforwardly talk about them without sounding salesly? We’re kicking off 2…

1
Designing Data Platforms For Fintech Companies 47:56

4M ago47:56

47:56

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Announcements Hello and welcome to the Data Engineer…

1
Rama with Nathan Marz 45:06

4M ago45:06

45:06

Building scalable software applications can be complex and typically requires dozens of different tools. The engineering often involves handling many arcane tasks that are distant from actual application logic. In addition, a lack of a cohesive model for building applications can lead to substantial engineering costs. Nathan Marz is the creator of …

1
Bonus Episode: SurrealDB with Tobie Morgan Hitchcock 57:17

4M ago57:17

57:17

SurrealDB is the result of a long-time collaboration between brothers Tobie and Jaime Morgan Hitchcock. The project has modest origins and started merely to support other projects the brothers were working on. However, over time the project grew and in 2021 they started working on it full-time. Since then the project has gained serious adoption. Wh…

1
Troubleshooting Kafka In Production 1:14:43

4M ago1:14:43

1:14:43

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Operating it at scale, however, is notoriously challenging. Elad Eldor has experienced these challenges first-hand, leading to his work writing the book "Kafka: : Troubleshooting in Production". In this episode he hi…

1
Adding An Easy Mode For The Modern Data Stack With 5X 56:12

4M ago56:12

56:12

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending all of their engineering effort on integrating systems that weren't designed with compatible user experiences. The team at 5X unders…

1
Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack 51:17

4M ago51:17

51:17

Summary If your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical dashboard isn't right. Unfortunately, it can often be complex or expensive to incorporate anomaly detection into your data platform. And…

1
2023 re:Invent Data Recap with Alex DeBrie 39:29

5M ago39:29

39:29

Alex Debrie and Bob Haffner recap their favorite announcements from 2023 re:Invent #data #dataengineering #aws Connect with AlexTwitter: @alexbdebrieBlog: alexdebrie.comBook: dynamodbbook.comPodcast: youtube.com/@SoftwareHuddle Connect with BobTwitter - @bobhaffnerLinkedIn - linkedin.com/in/bobhaffner Alex’s talkhttps://www.youtube.com/watch?v=PVUo…

1
Tracking Drug Smugglers and Migrating Databases with Benny Keinan and Lior Resisi 50:40

5M ago50:40

50:40

Maritime logistics is the process organizing the movement of goods across the ocean. Historically, this has been a challenging problem because of the multinational nature of shipping, as well as piracy, smuggling, and legacy technology. It’s also profoundly important for security reasons, and because 90% of what we buy travels over the oceans. Ocea…

1
Designing Data Transfer Systems That Scale 1:03:57

5M ago1:03:57

1:03:57

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Andrei Tserakhau has dedicated his careeer to this problem, and in …

1
The Right to Be Forgotten with Gal Ringel 47:45

5M ago47:45

47:45

Data breaches at major companies are so now common that they hardly make the news. The Wikipedia page on data breaches lists over 350 between 2004 and 2023. The Equifax breach in 2017 was especially notable because over 160 million records were leaked, and much of the data was acquired by Equifax without individuals’ knowledge or consent. Data brea…

1
Sofascore with Josip Stuhli 49:48

5M ago49:48

49:48

If you’re a sports fan and like to track sports statistics and results, you’ve probably heard of Sofascore. The website started in 2010 and ran on a modest single server. It now has 25 million monthly active users, covers 20 different sports, 11,000 leagues and tournaments, and is available in over 30 languages. Josip Stuhli has been with Sofascore…

1
Addressing The Challenges Of Component Integration In Data Platform Architectures 29:42

5M ago29:42

29:42

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. In this episode Tobias Macey shares his thoughts on the challenges that he is faci…

1
Daytona with Ivan Burazin 47:50

5M ago47:50

47:50

Cloud-based software development platforms such as GitHub Codespaces continue to grow in popularity. These platforms are attractive to enterprise organizations because they can be managed centrally with security controls. However, many, if not most, developers prefer a local IDE. Daytona is aiming to bridge that gap. It’s a layer between a local ID…

1
Transitioning from software engineering to data engineering 29:48

5M ago29:48

29:48

Every data team should have at least one data engineer with a software engineering background. This time on The Data Engineering Show, Xiaoxu Gao is an inspiring Python and data engineering expert with 10.6K followers on Medium. She’s a data engineer at Adyen with a software engineering background, and she met the bros to talk about why both softwa…

1
GraphAware with Luanne Misquitta 57:51

5M ago57:51

57:51

Knowledge graphs are an intuitive way to define relationships between objects, events, situations, and concepts. Their ability to encode this information makes them an attractive database paradigm. Hume is a graph-based analysis solution developed by GraphAware. It represents data as a network of interconnected entities and provides analysis capabi…

1
Unlocking Your dbt Projects With Practical Advice For Practitioners 1:16:04

5M ago1:16:04

1:16:04

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. While it is easy to adopt, there are many potential pitfalls. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects…

1
Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine 1:07:52

5M ago1:07:52

1:07:52

Summary Software development involves an interesting balance of creativity and repetition of patterns. Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. In this episode Eran Yaha…

1
Chronosphere with Martin Mao 48:08

6M ago48:08

48:08

Observability software helps teams to actively monitor and debug their systems, and these tools are increasingly vital in DevOps. However, it’s not uncommon for the volume of observability data to exceed the amount of actual business data. This creates two challenges – how to analyze the large stream of observability data, and how to keep down the …

1
Shining Some Light In The Black Box Of PostgreSQL Performance 54:51

6M ago54:51

54:51

Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. In this episode Lukas Fittl shares some hard-won wisdom about the causes and solution of many performance bottlenecks and the work that he is doi…

1
Surveying The Market Of Database Products 47:12

6M ago47:12

47:12

Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she…

1
Streamlit with Amanda Kelly 47:06

6M ago47:06

47:06

The importance of data teams is undeniable. Most companies today use data to drive decision-making on anything from software feature development to product strategy, hiring and marketing. In some companies data is the product, which can make data teams even more vital. But there’s a common problem – analyzing data is hard and time consuming. Lots o…

1
Defining A Strategy For Your Data Products 1:03:50

6M ago1:03:50

1:03:50

Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for information. In this episode Ranjith Raghunath shares his thoughts on how to build a strategy for the develop…

1
Modern Web Scraping with Erez Naveh 57:29

6M ago57:29

57:29

Today it’s estimated there are over 1 billion websites on the internet. Much of this content is optimized to be viewed by human eyes, not consumed by machines. However, creating systems to automatically parse and structure the web greatly extends its utility, and paves the way for innovative solutions and applications. The industry of web scraping …

1
Reducing The Barrier To Entry For Building Stream Processing Applications With Decodable 1:08:28

6M ago1:08:28

1:08:28

Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems …

1
Observability with Eduardo Silva 44:47

6M ago44:47

44:47

There are hundreds of observability companies out there, and many ways to think about observability, such as application performance monitoring, server monitoring, and tracing. In a production application, multiple tools are often needed to get proper visibility on the application. This creates some challenges. Applications can produce lots of diff…

1
Using Data To Illuminate The Intentionally Opaque Insurance Industry 51:58

7M ago51:58

51:58

Summary The insurance industry is notoriously opaque and hard to navigate. Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. In this episode he shares his journey of data collection and analysis and the challenges of automating an intentionally manual industry. Announcements He…

1
AI and Business Analytics with John Adams 30:03

7M ago30:03

30:03

It’s now clear that the adoption of AI will continue to increase, with nearly every industry working to rapidly incorporate it into their systems and applications to provide greater value to their users. Business analytics is a key domain that promises to be radically reshaped by AI. Alembic is an AI platform that integrates web data, product conve…

1
Vin Vashishta explains why we should stop using dashboards 35:45

7M ago35:45

35:45

Vin Vashista, the guy we all love to follow, has never seen a dashboard with positive ROI. This time on The Data Engineering Show, he met the bros to talk about the difference between BI dashboards and analytics that actually introduce knowledge. It’s no longer just about the data volume, it’s about quality and relevance.…

พอดคาสต์ที่ควรค่าแก่การฟัง

Data Engineering พอดคาสต์

พอดคาสต์ที่ควรค่าแก่การฟัง

คู่มืออ้างอิงด่วน