Data Quality

“Garbage in, garbage out” is a surprisingly common and a very costly truism. Poor data quality is the number one reason why business intelligence (BI) projects fail. A project team that does not address the problems of data quality head on is bound to end up with a solution that does not deliver reliable information. Cool visualization tools become useless if business users do not trust the data.

High costs of low quality
It is intriguing how often some form of the following situation exists in organizations, which already have a BI solution implemented.

All too often company's management and regulatory reports are actually based on manually edited spreadsheets. It takes a great amount of re-work to reconcile and correct the data every time a report needs to be produced. The semi-manual process involves extracting the data and getting it into a reporting tool only to realize that the data was incorrect. This causes the iterative loop of re-extraction from the sources and repeated analysis. Once a satisfactory set of source data is obtained the next phase begins. The data gets exported to spreadsheets numerous times, reconciled to conflicting departmental reports and manually fixed. In total, a lot of time and effort is wasted across an entire company because of redundant and inconsistent data.

Time and resources spent on the information rework are not the highest price to pay for poor data architecture. Nonquality data results in inaccurate view of business performance and leads to wrong or delayed decisions. Problems with data quality may also lead to regulatory non-compliance and bring in penalties. The effect on company’s bottom line may be millions of dollars in lost sales and cost savings opportunities. History knows extreme cases of companies who forgot that data is a major corporate asset needing proper management and went out of business as a consequence.

Improving data quality
How can companies protect themselves from data errors and achieve successful business intelligence? The problem isn’t easy to resolve, especially for organizations with high data volumes or very complex data. Success in this area starts with an understanding that achieving information quality is a continuous process rather than a one-time project. Practice shows that lasting results are achieved by organizations who implement data quality programs using a well-designed data governance framework. At the very least, a mature data governance framework should encompass the functions displayed in the depicted diagram.

Executed properly, data governance goes beyond policies, standards and processes. It ties tightly business and technology. The result is an increased speed of organizational response to the rapidly changing market demands.

Governance integrates a company’s investments in technology and data. It enables wise investments into enterprise strength architectures. This is much more effective and a lot cheaper than the practice of spending money on redundant databases and moving the replicated data around.

Master Data Management program is a critical element of successful data governance. There is a significant bidirectional influence between master data management and data quality.

Master Data Management (MDM)
The issues of poor data quality are often related to master data, i.e. data about customers, products, locations, etc. Conflicting and duplicate versions of data objects, such as customers and products, may exist across disparate or distributed enterprise systems. Managing a company’s master data requires that variant representations of the common data objects are consolidated and resolved into a single “best record”. This is a challenging process as the data spans multiple applications and business units. The problem of reconciling and consolidating master data is most acute within companies that went through reorganizations or mergers and acquisitions. To illustrate, consider some common challenges:

Successful MDM program relies on a collection of practices and technologies to create a common and consistent view of enterprise master data. High quality master data is subsequently fed to transactional and analytic applications. MDM technical solutions must possess the following key capabilities:

• inventory and identification of candidate master data objects
• resolution of semantics, hierarchies and relationships for master entities
• standardized information extraction, sharing and delivery
• a migration process for consolidating the “best records” for the master repository

How Lingaro makes a difference
While data quality is one of the most important aspects of business intelligence, it can also be one of the most frustrating. Lingaro consultants provide extensive knowledge that goes well beyond traditional data profiling and cleansing. We apply our experience to your project right from the start so that data quality can be addressed early on. A proactive approach to quality is much more cost-effective than fixing issues on the data warehouse side. We believe that quality should be designed into work-processes and applied at the level of transactional systems. Such an approach ensures that the analysis and reports remain synchronized with the source data.

There is no single data quality solution that fits all companies and problems. Lingaro will work with your team to plan and implement business processes customized to your company’s specific needs. The next step will be to design solution architecture. Finally, our consultants will help you select the right tools, or make better use of tools you may already have. The key is to take an incremental approach to ensuring quality which delivers benefits early and often. The result will be Business Intelligence solutions that meet business expectations in terms of timeliness, accuracy, completeness and above all confidence.