Data integration and migration challenges arise every time an organisation moves to a new data system, or wishes to combine multiple data systems, either internally or as a result of a merger, says Gary Allemann, MD at Master Data Management.
This is a significant undertaking, which, if it fails to meet expectations, inhibits a business’ ability to function effectively. Yet, surveys indicate that more than 80% of all data integration projects fail.
There are several reasons for this failure, but the most commonly cited issue is a lack of understanding of the extent and nature of problems that were to be experienced within the data itself. Data quality is critical to the success of any data integration and migration initiative, and organisations leave this aspect to the last minute at their peril.
Planning and data quality go hand in hand when it comes to data integration and migration, and any other big technology project for that matter. However, the moving of data is often an afterthought, since the business challenge revolves around implementing the new system or merging two disparate systems together.
Data is also often outside of the scope of most system integrators, and is deemed to be the responsibility of the business once the system has been implemented.
The issue that arises is that projects do not adequately plan for data migration or integration from the start, and it is common for them to get caught out by unexpected data issues as a result. In a typical data integration or migration scenario, during the initial project planning and scoping, data problems remain unidentified.
This frequently then requires a rework at a later stage in the project implementation, if issues are picked up during testing for example, which extends project deadlines and leads to unanticipated expenses. In fact, addressing these unplanned data issues can consume up to 70% of the time and budget in a rework later on, which results in costly and unnecessary delays.
The challenge with not planning for data at the outset is that systems are typically designed based on what people think the data should look like, and not what the data is actually like.
Issues such as invalid date information, missing codes, incorrectly captured information and duplicate fields may seem like small fry compared to the other challenges of implementing a new system, but the reality is that these data quality problems can have a significant effect on the functionality of the entire system.
Data issues can cause loads to be dropped when the data is transferred to the new system, which means that information will be missing. This can also lead to multiple duplicates being created, which can destroy the integrity of the database, take up unnecessary storage space and impact negatively on system performance.
Without information, businesses cannot function, and without ensuring data quality before a migration or integration, organisations cannot guarantee the availability of quality data, severely affecting business performance.
The biggest challenge around data integration and migration is proper planning from the start. This is exacerbated by large data volumes, which can make identifying problems a complex and time-consuming task.
The best way of avoiding these issues is to use effective data profiling techniques, which will enable a sound understanding of any data quality issues. It is also advisable to conduct a proper analysis and scope of the implications of the data take-on at an early stage in project implementation, so that data issues can be planned and budgeted for.
Businesses often underestimate the problems that data issues can cause, and the impact of amalgamating data or moving it over to a new system. This aspect is often overlooked in favour of the functionality of the system, but the reality is that data quality affects functionality.
Data quality remains one of the major reasons why IT projects fail to be delivered on time and on budget. High quality data lowers integration costs and project risks, increases user acceptance of new systems, and ensures systems meet the requirements of business users, helping to ensure a successful and problem-free project implementation.
