/*Blog module replace 'read more' text*/

Data migration experiences in building a bank’s Customer Master Database - Part 2

MINDSPIRE BLOG

Follow MINDSPIRE on social!

Mar 10, 2024 | Data Migration Blog

Our company, MINDSPIRE Consulting, has gained practical experience in many data migration projects during the last decade.

In our post along with our multi-year experience at a large Hungarian bank we review the typical tasks, challenges and lessons learned that we have encountered.

In the first part of our series of articles, we reviewed our banking data migration activities carried out during the first period.

In the second part we now cover our latest activities and summarize the lessons learned.

Duplication filtering of data included in the Customer Master Database

As data from the integrated banking satellite systems were continuously entered into the Customer Master Database, each time it was necessary to verify the purity and coherence of the data.

The most important element of this is the verification of the uniqueness of the customers, i.e. duplication checking, for which the solution hosting the Customer Master Database runs a number of validations during data loading.

For example, for document identifiers, a basic check is that two customers can not have the same data. But it is also necessary to check whether the customer to be migrated already exists in the Customer Master Database.

So, duplication checks have to be performed at several levels, at the source system, at the relationship between source and target systems, at the customer level and also for the customer data.

Duplication checks must be performed at the planning phase of banking data migration projects, within the source system, in terms of the source and target systems, otherwise the fields of the two systems cannot be matched and thus the logical mapping cannot be done.

Kamilla Szerletics

Kamilla Szerletics

Banking Data Migration Expert

Kamilla has more than 10 years of experience in the financial sector, with a proven history of working with data migration projects, including data migration tool developments.

She is involved in the full data migration cycle, including planning, coordinating migration processes, mapping source and target systems, maintaining migration databases, functional testing, and supporting UAT activities.

Besides working as a banking data migration expert, Kamilla is managing banking data migration projects since 2019, and acts a product manager for MINDSPIRE DELTA Banking Data Migration Tool.

In addition, these checks have to be performed in several views, because different verifications must be performed if the customer already exists in the bank’s Customer Master Database or if it does not exist yet and has to be created as a new customer.

In the case of an existing customer, we are talking about data enrichment in the target system, which can be broken down into further migration processes from a data migration point of view. A distinction has to be made between the case where the individual or business to be loaded from the source system in the bank’s Customer Master Database has not yet been created as an actual customer, but only as an lead and only has such a role created there. In such cases, the first step is to convert him or her into a customer in the target system and to assign him or her the corresponding new role and data.

Accordingly, during each data migration, multiple logical mappings and processes must be established for each subset. Thus, when loading a given source system, the same tables are loaded in the target system, but according to different rules and in several steps.

In some cases, the process may be further complicated by functional constraints in the bank’s Customer Master Database. Here, certain data, such as the document number of a customer, cannot be modified by mass migration, and therefore deletion does not work. It is possible that for customers in both systems, the document data in the source system may be more recent than in the target system. In such cases, however, it is not possible to overwrite this data during migration, so the affected items must be corrected manually afterwards.

Our migration methodology and the MINDSPIRE DELTA data migration tool can fully analyze data from multiple systems, report on different business needs, then develop duplication detection algorithms, apply different migration procedures to manage duplicate customers.

We perform duplication searches between and within systems according to the capabilities of the bank and the target system. MINDSPIRE performs the detection of customer duplications and the assignment of the customers considered as primary along different algorithms and visualizes the results in reports and statistics for the bank. The result of the duplicate search is manually verified and approved by the bank’s operational areas, and our company utilizes the final duplicate customer lists and reports during the transformation of the migration data. The tasks associated with the duplication search are perhaps the most challenging of all customer migration projects, as it is perhaps the most sensitive data set for the bank.

Challenges related to providing data-cleanliness

In addition to verifying the uniqueness of customers and data, the data migration of each banking satellite system first required preparing the information in the source system for compliance with other validations in the Customer Master Database solution. The process of preparation consists of checks, development of error reports and then corrections based on the error reports.

These source system checks are not performed in the Customer Master Database validation solution but using the MINDSPIRE DELTA data migration tool, because data can only be imported into the Customer Master Database solution if all elements of the data comply with all validations contained therein. So, a customer could only be migrated to the target system if all the data associated with it met the validations. Otherwise, not only was the transfer of the incorrect data concerned unsuccessful, but the customer itself was not created in the Customer Master Database, and accordingly, none of its other data was transferred to the CMDB, which was reported by the system in the form of an error message.

Moreover, in such cases, the Customer Master Database stops loading the data for the given customer at the first incorrect data, so that no further data is checked, and the error message will only contain the message for the first problematic field. So, if a customer’s data contains multiple errors, relying on the built-in checks in the system, these can only be detected in multiple rounds, after the error detected in that step has been corrected, on the next loading attempt, which would result in a rather long and frustrating data cleansing process for the parties involved.

These SQL checks must be performed at field level, between fields and per data field, and the aggregated results of the preliminary checks are summarized in an error report, which provides an overview of the data to be migrated and how well it meets the data cleanliness criteria.

The error reports contain a number of elementary problems, such as the fact that a few hundred customer names cannot be migrated to the Customer Master Database because they contain characters that are not allowed there, such as umlauts or commas. They may also identify discrepancies or gaps based on certain contexts.

Some of these error lists were forwarded to the responsible persons in the respective source system, who manually modified the incorrect data in the satellite system.

In other cases, it was also possible to define transformation rules to be implemented during the migration, which avoided the need for manual correction. Examples included replacing multiple spaces with a single space and capitalizing lowercase letters at the beginning of a name.

A crucial part of the migration process to be developed in a given project is the implementation of data cleanliness, and our experts played a significant role in this, as data errors can be identified in an automated process without error reports based on a complete analysis in our MINDSPIRE DELTA data migration tool and the target system requirements.

Bank managers already receive ready-to-use error reports for datasets and error-cases, facilitating manual corrections in the source system with additional reports.

Furthermore, for errors affecting larger data volumes, we always find a solution, a transformation rule for automatic correction in the migration tool, when we can already deliver the data for loading corrected in the target system migration file.

Manual modification of the bank’s customer data

As can be seen from the above, in addition to mass data migration, manual data modification is required in many cases because not all checks or corrections can be automated.

These manual modification activities are supported by providing bank staff with the verifications generated in the MINDSPIRE DELTA data migration tool, error lists, questionable items and correct data that could not be loaded during the migration.

Delta Tool data transfer Erste 2022

The importance of different customer roles and their impact on data migration

Customers created in the bank’s satellite systems may have different roles, the management of which is another fundamental purpose and task of the Customer Master Database.

By default, each customer is assigned a so-called generic role, for which only a few fields are required, such as name and a document number, or company name and a company identification code. However, in addition to this, if a customer is to be given a role by the administrator, for example as a bank account holder, the Customer Master Database will also expect additional data to be recorded, whereas for example a contact person will require other information. Thus, there are field obligations for the different possible roles, which need to be taken into account during both data capture and data migration.

For customers with multiple roles, the data needs of all of them must be taken into account, but care must also be taken to ensure that certain roles are mutually exclusive. For example, a customer cannot be both an occasional customer and a bank account holder, and roles may have an expiry date.

These complex requirements and interrelationships between roles must also be met before any data migration, in order to ensure that the customer is correctly populated in the Customer Master Database.

In the event that a particular customer cannot be created in the role it would otherwise have in the Customer Master Database because of incomplete data, this should be indicated by so-called desired roles. In such cases, the customer will then have to provide the missing data in order to be able to fill this role in the future.

Temporary disablement of the data validation rules

There were also cases when it was clear that the data in the source system could not be loaded into the bank’s Customer Master Database because it did not comply with the validation rules.

In such cases, it was possible to disable one or more of the strict data validation rules in the target system for the duration of the migration, allowing for back-loading.

Of course, in such cases, when clients are loaded in this way, administrators will encounter the warning during live use until the data problem is resolved.

Our experience in managing and organizing data migration projects in banks

Defining the project scope

Based on our practical experience, in most cases, the scope of our bank data migration projects has been continuously expanded during implementation.

As it was not always possible to fully assess the complexity of the data stored in the system at the design stage or to define the requirements for the functionality to be developed later, the needs were developed during execution.

For example, the importance and resource requirements of identifying duplicates as a task were not clear at the initial stage, but the experience collected has made clear to all participants the importance and technical complexity of this task. As a result, more time was subsequently allocated by the management to the planning and execution of duplication identification procedures in each of the smaller customer data migration projects.

In addition, the planning phase of the data migration process has been given more attention, with a more thorough analysis of the issues, allowing for a more accurate scope and more realistic timelines for the execution phase.

Cooperation between the bank and data migration experts

During the execution of data migration activities, it is not always possible to clearly delineate the responsibilities of the parties involved, even if their roles are clearly defined at the outset of the project.

The specificities of the banking data migration activities require a very close cooperation between the bank and the external data migration expert, which leads to a high degree of interdependence. If one of the parties fails to perform its tasks on time or accurately, the whole project is affected and errors often only become apparent at a later stage, when it is not easy to establish responsibility.

Core banking system parallel changes illustration

The overlapping of tasks and responsibilities is not only observed at the level of execution, but also at the level of task management. MINDSPIRE Consulting’s data migration team in banks is usually led by one or two professional managers, who coordinate and harmonize tasks and communicate directly with the bank managers. He or she is the person who coordinates and relays the bank’s requirements to our experts in the form of data migration tasks.

The client typically delegates a project manager responsible for the data migration and a test manager who coordinates with the bank’s employees and forwards the tasks we have given them, such as processing the aforementioned source system error reports, testing the data migration, or performing post-migration tasks. However, despite this, in many cases the tasks of the two teams’ designated professional managers and liaisons are highly intertwined, so interdependence and close cooperation can be observed here as well.

Remote work on banking data migration projects

For a while, COVID-19 significantly altered the way we worked, which had a profound impact on our then ongoing data migration projects. We have previously explored this topic in two posts, which can be accessed here:

Data migration is essentially a team effort, where coordination of tasks is essential. Although the tasks of individual experts can usually be delineated by data domain, each step in the overall data migration process is interlinked with the other activities.

For this reason, we preferred to work face-to-face throughout our data migration projects, so even during the epidemic our experts typically worked together in the same office.

However, in general, w had less and less opportunity to meet and interact with clients in person, which in our experience had a minor but negative impact on the effectiveness of our work.

MINDSPIRE’s contribution to banking data migration tasks

For this specific project, our company has delegated a 3-4 person banking data migration team, led by a professional manager who is mainly responsible for coordinating the team, planning the data migration process, and channeling the client needs.

Over the last six years, nine systems and datasets have been migrated to the Customer Master Database. In addition, we have completed the planning phase for several systems to be included and are in the process of preparing for the inclusion of additional applications.

In connection with data migration activities, several data cleansing projects have been carried out to ensure data consistency and data cleansing of the systems linked to the Customer Master Database.

We are currently working on a longer data cleansing project to correct data errors arising from functional synchronization errors after the implementation of the Customer Master Database, by examining customer data in the bank’s main systems.

In this process, our banking data migration experts will generate error reports covering data inconsistencies between systems and the error events that occur. We then develop migration datasets that implement specific data enhancements and perform a go-live in a phased approach similar to migration projects, with weekend transition tasks.

Benefits of banking data migration services provided by MINDSPIRE Consulting

Our MINDSPIRE DELTA data migration tool and our methodology, based on many years of migration experience, allow us to process the data to be migrated efficiently and completely. As described above, the process includes the following main activities as a standard case: source system data analysis, generation of error reports, support for data cleansing, implementation of data transformations taking into account the needs of all systems involved.

After the data migration has been completed, a comprehensive summary, called “Release notes”, is also produced to make the results understandable to other business areas of the bank. This includes a number of useful documents such as bug reports, task lists and statistics on the migration process.

Each step of the data migration and the process described above can be technically traced back in the closed database of the MINDSPIRE DELTA data migration application, from which our colleagues can also retrieve any requests or reporting needs that may arise.

Do you have any questions about our banking data migration services? Contact us now and our experts will answer it!

Please see MINDSPIRE’s related:

 

Data Migration
Services

Data migration services icon 2022

MINDSPIRE DELTA
Data Migration Tool

Data migration experiences in building a bank’s Customer Master Database – Part 2
Project management contact form

Do you have a question or comment about this post?

Send us your message and our experts will contact you!

Data migration icon

Our latest Data Migration References

Are you interested in our Banking Data Migration services?

You can find more information here:

Share This