Data Mesh: A Deeper Dive

Thoughtful Software Podcast

In this episode, Andrew and Fahad take a deeper dive into Data Mesh Architecture, discussing why it's important as well as examining the value of centralized vs decentralized structures. They also discuss data mesh nodes, examine some common issues a business might face with data mesh, and explore ways an enterprise can transition from a centralized structure to a data mesh.
SHOW NOTES:
What is data mesh and why is it important?

• As opposed to a data lake, which is the attempt to centralize into one system many different pools of data, a data mesh decentralizes the data but centralizes its governance.
• It is important because it helps data-driven organizations function in a way that is sustainable.
• Four issues at work that are slowing businesses down:
  1. Decentralized ownership of systems within a centralized structure
  2. Business is moving faster than ever and centralized structures can't keep up.
  3. Data lakes have become data oceans, impossible to index and understand.
  4. Some data is in silos and not accessible to other departments

Centralized vs Decentralized

• There is no bad architecture - centralized, decentralized, monolith, distributed, monolith microservice etc. The only right architecture is one that gets the job done.
• Data lake docks are a piece of business logic that prevents the system from changing. Pulling away from the doc and allowing the business logic to where it was is created then will allow the business to move much faster.
• Data mesh allows a specific department to own itself without the entanglement of depending on other departments for their data.
• Smaller organizations may be better off starting with a data lake, but build your mesh as you grow and become more complex.

Data Mesh Nodes

• Bigger companies that are acquiring smaller companies can incorporate their data with a mesh node.
• You can still have data pipelines that merge multiple mesh nodes that create specialized data ponds which have only the info they need to do their job and help their department.
• Data mesh nodes enable you to think about the data to how to compose that data to solve a problem.

Issues Concerning Data Mesh

• Overdeployment, not every company is big enough.
• Lack of overall governance can cause data mesh nodes to become swampy
• Companies are going to make nodes too big.
• It is 1/4 of the size of the data lake market but will likely surpass the data lake market in the next five years as data mesh picks up the failures of data lakes.

How Does an Enterprise Roll Out Data Mesh?

• Start at your biggest pain point and work out from there, pulling the data apart piece by piece until eventually, you are decentralized.

RESOURCES MENTIONED:

Kubernetes
Tableau
Salesforce
NetSuite
Snowflake
PREVIOUS EPISODES MENTIONED:

Episode 45 - The Evolution of Data Architecture: Moving to a Data Mesh
Episode 51 - Microservices: Trading Code Complexity With Organizational Complexity
Follow us @fahsho12 and @andrewwwolfe and share your insights and questions with #thoughtfulsoftware.