By: A Staff Writer
Updated on: Aug 11, 2023
The following is a comprehensive whitepaper on the issues of communication and Change Management in Data and Analytics.
In today’s data-driven world, the efficient management of data is paramount to the success of any organization. However, a growing problem plaguing the industry is the lack of communication and ineffective change management between different stakeholders, especially the owners of production systems and the owners of analytical systems.
Owners of production systems often have no understanding of where their data ends up, who is using it, and for what purpose. Conversely, those who consume the data are typically unaware of when changes are made, who made them, and why. This disconnect leads to chaos, confusion, and a lack of trust in the data’s validity.
A survey by Experian in 2020 found that 95% of organizations see negative impacts on their operations due to poor data quality, which directly correlates with the lack of proper communication and change management protocols.
This communication gap doesn’t only affect one side of the equation; it’s a problem that has significant implications for both production and analytical systems.
The result is a fragile and unreliable data environment where engineers scramble to fix problems after they’ve already occurred. Often, it’s too late, and the trust in the data has been irreparably damaged.
Given these persistent issues, there’s a clear and urgent need for a new paradigm in data change management and communication. A more transparent, proactive, and collaborative approach could foster better alignment between data producers and consumers, minimize misunderstandings, and enable a more robust and resilient data ecosystem.
Some innovative companies have already begun to invest in this area, creating platforms and tools that provide real-time notifications of data changes, collaborative environments for discussing data requirements, and clear documentation of data usage and expectations.
However, the industry as a whole must move toward this new paradigm. It’s not just about new tools and technologies; it’s about a fundamental shift in how data stakeholders interact and communicate. Without this change, the burgeoning field of data analytics risks becoming bogged down in confusion, errors, and mistrust.
Communication and change management in data and analytics are not merely technical issues; they are critical business challenges that require immediate attention. The current gaps in communication and change management are leading to serious implications for both production and analytical systems. Embracing a new paradigm that promotes transparency, collaboration, and understanding between all data stakeholders is not just a wise strategy—it’s an essential one.
Data management has undergone a dramatic transformation over the last few decades. In the early days of computing, data was often siloed, stored in isolated databases with little interoperability. As businesses grew and technology evolved, there was a need to integrate disparate data sources and facilitate access across various platforms.
The introduction of cloud computing and Big Data technologies allowed organizations to manage massive amounts of data more efficiently. However, this growth also brought complexity, making it challenging to maintain consistency, accuracy, and timeliness of information. According to Gartner, by 2022, 60% of organizations have had to modernize their data quality solutions to respond to these growing challenges.
Data engineers play a crucial role in creating and maintaining the architectures, pipelines, and data sets that make information accessible and useful. They are the bridge between raw data and analytical insights, ensuring that data is clean, well-structured, and ready for analysis.
A data engineer’s responsibilities have expanded from mere technical tasks to encompassing communication and alignment with various stakeholders within the organization. They must now understand not just how data is used but why and by whom, necessitating a broader skill set and closer collaboration with business users, analysts, and software engineers.
The efficient flow of accurate information between data producers and consumers is vital for making informed decisions. Without clear communication, there is a risk of misalignment, leading to incorrect analyses and misguided strategies.
For example, if a marketing team is unaware of changes made to customer segmentation by the data team, their campaigns may target the wrong audience, wasting resources and missing opportunities. A 2018 report from Harvard Business Review highlighted that companies with strong data alignment between departments were three times more likely to report significant improvements in decision-making.
Data debt refers to the “cost” incurred when shortcuts are taken in data management, often as a result of quick fixes or temporary solutions. These might include duplicated data, inconsistent naming conventions, or lack of documentation, leading to confusion and additional work down the line.
Like financial debt, data debt can compound over time if not addressed, making future changes more time-consuming and expensive. A survey by Deloitte found that 62% of organizations see data quality issues as an obstacle to their digital transformation efforts, largely stemming from accumulated data debt.
As businesses grow, the scale and complexity of data management increase exponentially. More data sources, more users, and more demands for insights mean that traditional approaches may no longer be sufficient.
A small start-up may be able to manage data with a few spreadsheets, but a multinational corporation dealing with petabytes of data across different geographies will need sophisticated tools, processes, and governance. The ratio of data engineers to software engineers can become skewed in larger organizations, further complicating change management. Gartner estimates that by 2025, 70% of organizations will have difficulties scaling their data management solutions to meet future business needs without significant investments in their data management practices.
Understanding the background and context of data change management and communication requires recognizing the evolution of data management, the expanding role of data engineers, the vital connection between data producers and consumers, the concept of data debt, and the profound impact of the business scale. Addressing these complexities calls for a concerted effort across the organization, leveraging both technology and human collaboration, to build a data management system that is robust, agile, and responsive to the ever-changing business landscape.
Data producers are responsible for creating, storing, and managing data. However, they often operate in isolation, without fully understanding the downstream needs or applications of the data they produce. This siloed approach leads to inconsistencies and misunderstandings. For example, a slight alteration in a data format may seem trivial to the producer but can have a substantial impact on the consumer’s ability to use that data effectively.
Data consumers, on the other hand, rely on data for analysis, reporting, and decision-making but are often kept in the dark about the underlying changes or the rationale behind them. Without clear communication channels with the producers, they may find themselves grappling with unexpected changes that hamper their ability to deliver timely and accurate insights.
In a 2019 survey by NewVantage Partners, only 24% of executives reported that their organizations were data-driven, citing lack of organizational alignment and collaboration as major barriers.
Breaking changes refer to alterations in data that disrupt existing systems, tools, or processes. Such changes can lead to significant problems, such as:
Similarly, new data availability without proper communication can lead to missed opportunities for leveraging fresh insights, creating a competitive disadvantage.
Data engineers find themselves at the crossroads of these challenges, tasked with managing change amid a lack of communication between producers and consumers. Their hurdles include:
A study by Syncsort found that 60% of data professionals spend half or more of their time on data quality and data preparation rather than on analytics or deriving value from the data.
The principle of Garbage In, Garbage Out refers to the quality of output being determined by the quality of the input. In the context of data management, if incorrect or poor-quality data is fed into the system, the resulting analyses, reports, and decisions will be flawed.
In many organizations, the ratio of data engineers to software engineers is disproportionately low. This imbalance leads to the following:
Forrester Research indicates that only 28% of firms report success in balancing technical and business resources effectively, highlighting this critical challenge.
The complex problem of data change management and communication demands a multifaceted exploration. The lack of clear communication between data producers and consumers leads to numerous challenges and consequences that permeate the entire data ecosystem. Understanding these issues is not merely an academic exercise; it’s a business imperative. Addressing these problems requires a systematic approach that integrates technological solutions, fosters collaboration, and prioritizes quality at every stage of the data lifecycle.
As organizations expand and data volumes increase, the problems related to data change management and communication can quickly magnify, leading to far-reaching effects. The compounding nature of these issues, especially at scale, creates serious challenges that can impede an organization’s growth and success.
A lack of proper change management can result in an immediate drop in data quality. At scale, even small inconsistencies can lead to significant problems, such as:
A study by IBM estimated that poor data quality costs the U.S. economy $3.1 trillion annually, highlighting the economic impact of this problem.
Data debt, or the compounding costs associated with shortcuts and temporary fixes in data management, grows more problematic as the scale of operations increases.
Quick fixes or ‘hot fixes’ may solve immediate problems but often lead to longer-term issues:
Without proper governance, replicated tables and duplication become widespread:
A report from Veritas found that, on average, 52% of all data within organizations is considered “dark data,” meaning its value is unknown, largely due to these kinds of issues.
The entire data warehouse can become compromised:
Gartner predicts that by 2024, 50% of data governance initiatives will fail to deliver the desired business outcomes due to a lack of proper attention to data quality.
As problems multiply, data teams may find themselves trapped in a cycle of managing data debt:
According to a survey by Accenture, 79% of executives agree that organizations are basing their most critical systems and strategies on data, yet many have not invested in the capabilities to verify the truth within it.
The effects of data change management and communication issues at scale are far-reaching and profound. The immediate degradation of data quality, accumulation of data debt, impact on data warehouses, and shift in data team focus from solving business problems to managing data debt can hinder an organization’s ability to innovate, make informed decisions, and compete effectively. Addressing these challenges requires a concerted and strategic effort that places data quality and collaboration at the forefront of organizational priorities. By doing so, organizations can transform their data from a potential liability into a powerful asset that drives growth and success.
Addressing the multifaceted problems associated with data change management and communication requires a strategic approach. By implementing targeted strategies that emphasize collaboration, transparency, governance, and scalability, organizations can mitigate these challenges and transform their data management into a robust, efficient system.
Involving data engineers in the early stages of decision-making ensures that data considerations are an integral part of the process:
For example, companies like Spotify have emphasized collaboration between data engineers and business teams, leading to more coherent and efficient data strategies.
Establishing a structured framework for change management helps in coordinating and controlling changes across the organization:
A McKinsey report indicates that organizations with a well-defined change management strategy were 3.5 times more likely to outperform their peers.
A visibility system provides insights into how, where, and why data is used:
Tools like Apache Atlas and Collibra offer visibility into data lineage and governance, enabling organizations like eBay to manage complex data landscapes effectively.
As organizations grow, scaling the analytical environment becomes vital:
Forrester’s research shows that 56% of global data and analytics decision-makers have either implemented or are expanding their implementation of some form of automation in their data platform.
Preventing and mitigating the problems associated with data change management and communication requires thoughtful, strategic action. By engaging data engineers early in the decision-making process, creating a robust foundation for change management, implementing visibility systems, and employing techniques to scale the analytical environment, organizations can build resilient, effective data ecosystems. These strategies foster collaboration, enhance transparency, and ensure that data remains a valuable asset that supports innovation, informed decision-making, and competitive success.
Real-world examples provide valuable insights into the practical application of strategies for data change management and communication. By examining both successes and failures and identifying trends across various industries, we can gain a more nuanced understanding of what works, what doesn’t, and why.
G.E. implemented a highly collaborative approach that integrated data engineers, analysts, and business stakeholders. This approach led to:
American Express emphasized data quality across all levels of data management. By creating standardized processes and utilizing advanced data quality tools, they achieved the following:
The initial launch of Healthcare.gov faced significant challenges:
This case underscores the importance of a well-coordinated, governed approach to data management.
A well-known retail chain accumulated substantial data debt by neglecting proper change management:
This failure highlights the risks of neglecting data debt and emphasizes the need for proactive management.
Different industries present unique challenges and opportunities in data change management:
Case studies and real-world examples illuminate the complexities and nuances of data change management and communication. Success stories offer inspiration and a roadmap, while lessons from failures provide critical warnings and insights. The varied trends across different industries further enrich our understanding, emphasizing the need for tailored, strategic approaches. Together, these examples create a rich tapestry of evidence that informs and guides effective practice, underscoring the universality of the principles of collaboration, quality, governance, and adaptability in achieving data excellence.
The changing landscape of data management necessitates strategic, forward-thinking approaches to ensure continued effectiveness and alignment with business goals. Below are tailored recommendations for different aspects of data change management and communication that can guide future directions.
Data owners play a crucial role in managing data quality and facilitating change. They should adopt the following best practices:
Embracing technology and automation can lead to increased efficiency and consistency:
Effective collaboration and communication are at the heart of successful data change management:
Ensuring that data management aligns with the broader business strategy is paramount:
The future of data change management and communication lies in adopting a comprehensive and nuanced approach that blends best practices, leverages technology, fosters collaboration and aligns with business goals. By embracing these recommendations, organizations can build resilient, efficient, and effective data ecosystems that not only address current challenges but also adapt to future needs and opportunities. These strategies, grounded in real-world examples and insights, form a blueprint for success in an increasingly data-driven world. They emphasize the need for a dynamic, integrated approach that recognizes data not merely as a technical asset but as a vital component of business strategy, innovation, and growth.
The multifaceted challenges surrounding data change management and communication have emerged as critical issues for modern organizations. As data continues to play an increasingly vital role in business strategy and decision-making, the strategies and insights discussed in this paper provide both a roadmap and a clarion call to action.
The landscape of data management has evolved rapidly, revealing a series of interconnected problems:
Solutions must encompass the following:
A recurring theme across these solutions is the necessity of shifting from reactive to proactive management. The saying, “An ounce of prevention is worth a pound of cure,” aptly applies here. As demonstrated by the 20% reduction in time spent on data preparation by G.E. through a collaborative approach, the ability to anticipate and prevent issues before they arise is invaluable.
The complexities of data change management are not challenges to be faced by individual departments or silos within an organization. They require the coordinated effort of multiple stakeholders:
The task ahead for industry stakeholders is both daunting and inspiring. It demands innovation, collaboration, foresight, and commitment. Yet, the rewards are equally compelling. By adopting the principles and practices outlined in this paper, organizations can transform their data management from a source of confusion and cost into a wellspring of insight, agility, and value. The time for action is now. The path forward has been illuminated, and the future awaits those bold enough to take the necessary steps. The transformation of data change management and communication from a problem into an opportunity is not just a possibility; it’s imperative for success in the digital age.