The collection, integration, and governance of data remain challenging for businesses today. Data mesh, a democratized approach to data management, proposes a way out. In this article, Sashank Purighalla discusses how the approach and its implementation will develop in the future.
Originally published on ToolBox.com
More than ever, unlocking the data dividend is vital to business success, with 81% of leaders (according to a Melbourne Business School and Kearney impact assessment study in 2020) in data and analytics being more profitable than those lagging behind. While traditional, monolithic architecture and databases have not yielded the desired results, they have inspired the birth of an approach known as data mesh.
Since its inception in 2019, data mesh has become a key asset in company data management, shifting data ownership structures from data experts to domain leaders. With the understanding that data drives organizational value, the concept sees data as an asset geared towards intelligence and human use rather than only serving technical attributes.
Deloitte defines data mesh as “a democratized approach of managing data where different business domains operationalize their own data, backed by a central and self-service data infrastructure.” The four main pillars include data ownership, data as a product, data being available on a self-service platform, and governance through a federated computational model.
So, data mesh is increasingly seen as a form of data democracy. And just like any democracy, it brings undeniable benefits alongside a set of challenges to overcome. Let’s explore what’s ahead for data mesh and what new solutions and perspectives businesses must consider.
See More: In the Age of Multi-Cloud, Is Data Governance Receiving the Attention It Deserves?
Signs of a Maturing Regime
In 2021, data mesh stepped out of the theoretical dimension to bring tangible value through tools and setups. In 2022, it is set to become a more mature model that truly lives its data as a product principle with more clarity within organizations and more offerings in the market.
Today, advancing data empowerment is often left to the individual company or to the leader driving the initiative and is more directive than prescriptive. In healthcare, for example, there are clear standards for communication and data exchange that are valid for that entire regulated industry. But within other industries, or for data as a segment by itself, this doesn’t exist. In 2022, we will see an increasing need for the formation of such standards, with specific industries pioneering this faster than others.
The need to navigate these standards will mean that the demand for self-service platforms and easy-to-deploy solutions will increase. More providers will emerge, responding to the demand for data mesh structures that is likely to grow exponentially over the next three years.
The existing piece-meal solutions and DevOps best practices only cater to a specific technical need that a developer or engineer has, leading to the automation of merely a small aspect of the data ecosystem. Developing powerful end-to-end data mesh solutions will be critical for companies to build resilient and scalable architecture.
After all, solutions don’t come from automating one small thing. They come from recognizing that in today’s hyper-digital landscape, organizations can’t afford to waste resources on fractured automation and siloed applications – not even mid-sized companies. They can’t afford an outcome that isn’t consistent with the ever-changing regulatory environment either.
From the democracy analogy perspective, they need to have robust and stable frameworks that can operate effectively and have the power to enforce governance. That’s why, in the future, the focus will be on the automated platforms that can help establish resilient ecosystems and provide native governance that guides results.
See More: What Is Data Governance? Definition, Importance, and Best Practices
Avoiding Data Anarchy at All Costs
Anarchy comes in stark contrast to a functional democracy. A common challenge for data mesh is inter-domain chaos and data duplication, which originates from ineffective governance. Within self-service, solid rules and clear hierarchies are needed to reach an innovatively governed, healthy system. So, as more freedom means more responsibility, a greater focus on smooth governance will be a key priority for 2022.
Instead of facing the notorious rigidity of data warehouses or the limitless freedom of data lakes, data mesh offers a middle ground. It comes to an organization and says: “I will not give you rigid designs, nor an open store for whatever you want. I will give you a little bit of both.” But then, what do you do to enforce standards across multiple interoperable applications? With many small domain groups that each understand their data, how do you get them to follow the same standards and produce their data in a way that’s also consumable for everyone? The future is going to demand an enforcement paradigm that oils the entire wheel of data mesh to function seamlessly.
Whether it’s appointing data stewards that go in and audit in the real world or implementing digital auditing mechanisms with established guard rails, a data democracy needs its policing force. The human and machine forces will likely coexist in a symbiosis, although the balance between them will depend on a specific organization and its need. Some solutions are more expensive but get to work immediately – others are more cost-effective but require laborious implementation and regular readjustments.
Organizations will need to discover governance gaps within their systems in the future, and they will have to challenge this with new forms of enforcement. More businesses will also shift their architecture to adopt data as a product better. Moving down that pipeline, they will have to carefully assess their existing tools and adopt new ways of following regulatory and industry standards.
