Core Banking Data Migration Pipelines for Bank Conversions
Moving Decades of Records Without Losing a Single Balance
Core conversions are usually described as system replacements, and they are experienced as data events. The new platform can be perfectly configured and thoroughly tested, and if the migration loads a status code the target interprets differently, thousands of accounts behave incorrectly from the first hour. Logic defects get caught in testing because someone can reason about them. Data defects arrive as customer balances that are wrong, and they arrive at scale.
That asymmetry is why core banking data migration deserves to be engineered as a product rather than run as a project task. The pipeline needs to be repeatable, the reconciliation needs to be arithmetic rather than judgment, and every ambiguous decision needs a name attached to it.
Why does data migration decide the conversion?
Because the new system can only be as correct as what you loaded, and correcting it afterwards is a customer event.
| Failure | How it presents | Cost |
|---|---|---|
| Poor source quality carried forward | Wrong balances, failed processing | Customer-facing, immediate |
| Ambiguous mapping resolved by guess | Products behaving unexpectedly | Discovered weeks later |
| History gaps | Cannot reproduce statements or answer disputes | Regulatory and complaint exposure |
| Broken referential integrity | Orphaned accounts, missing relationships | Processing failures |
| Encoding and collation differences | Names corrupted, sorting changed | Reputational, hard to unwind |
| Derived values recomputed differently | Interest, accrual, and staging discrepancies | Financial restatement |
| In-flight items mishandled | Duplicate or lost payments | Immediate financial loss |
Why is cleaning data later a fatal plan?
Because after go-live a data fix becomes a customer-affecting adjustment.
Before migration, a wrong address or an incorrect accrual is a data defect you can correct silently. After migration it is a change to a live customer record with accounting entries, possible disclosure requirements, and an audit trail question about why it was wrong. That difference is why cleansing decisions must be made and executed before the load, and why the data quality work usually starts a year before the conversion date. The underlying constraint is the one described in this guide to improving data quality across legacy systems.
Is your data cleansing scheduled before the load or planned as post-conversion remediation?
What does the pipeline look like?
Eight repeatable stages, automated end to end so a full run can be executed on demand.
| Stage | Purpose | Output |
|---|---|---|
| Extract | Pull source data at a consistent point | Immutable extract with control totals |
| Profile | Measure actual content against expectations | Quality report, anomaly inventory |
| Cleanse | Apply agreed corrections | Cleansed set with change log |
| Map and transform | Convert to target structures and codes | Transformed set with rule versions |
| Validate | Check target business rules before loading | Rejection report by rule |
| Load | Insert into the target platform | Load statistics per object |
| Reconcile | Compare source and target arithmetically | Reconciliation report, differences |
| Verify and sign off | Business confirmation per domain | Signed evidence pack |
Why automate rather than script each run?
Because you will run it many times, and the value comes from repeatability.
A conversion involves multiple full-scale rehearsals plus the real event, and every run must be identical except for the input. Hand-run scripts produce variation nobody can account for, which destroys the evidence value of the rehearsals. Build the pipeline as versioned code with parameterised environments, log every rule version applied, and make a full run a single command. That also means a defect found in rehearsal four can be fixed and the entire run repeated in hours rather than weeks, which is what keeps the schedule intact.
How do you profile the source honestly?
By measuring what the data contains rather than what the documentation claims.
Profiling reliably finds nulls in fields the target requires, orphaned child records whose parents were purged years ago, duplicate customers under different identifiers, values outside any documented range, status codes that appear in production and in no specification, dates that cannot be real, and encoding artefacts from a platform migration two decades ago. Run profiling early and repeat it, because the source keeps changing while you work. Then publish the anomaly inventory with counts, since a defect affecting eleven accounts and one affecting four hundred thousand need different responses and both get called a data issue in status meetings.
Who decides what to fix versus carry forward?
A named business owner per domain, with the decision and rationale recorded.
Three responses exist for every anomaly: correct it before migration, carry it forward as-is, or exclude the record with agreed handling. Each is legitimate and each is a business decision with consequences, so the migration team should present options and impact rather than choose. Record the decision, the owner, and the date, because in the year after conversion somebody will ask why a population of accounts looks unusual and the answer needs to be a documented decision rather than an archaeological exercise. Duplicate customer records deserve particular attention, since merging them is a customer-visible act, as covered in this guide to solving duplicate customer records.
How do you handle mapping ambiguity?
Explicitly, with every unmapped or derived field owned and signed off.
Mapping is straightforward for perhaps eighty percent of fields and contentious for the remainder. The target may have no equivalent field, may require a value the source never captured, may define a status differently, or may need a derivation from several source fields. For each, choose deliberately between extending the target, deriving the value with a documented rule, defaulting with business approval, or dropping the field with sign-off. Never let a developer resolve an ambiguity quietly at build time, because that becomes undocumented behaviour in a brand new system, which is precisely what you were trying to escape.
What about derived and calculated values?
Recompute them in the target and reconcile against the source rather than migrating the number.
Accruals, delinquency stages, available balances, and risk classifications are outputs of logic. Migrating the stored value hides a difference in how the target computes it, so recompute in the target and compare with the source value as a reconciliation control. Differences then reveal either a mapping defect or a genuine behavioural difference requiring a business decision, which is exactly what you want to discover before go-live rather than in the first accrual cycle. That comparison discipline is the same one described in parallel runs and dual-ledger validation.
Are derived values being migrated as stored numbers or recomputed and reconciled?
Talk to Digiqt about mapping governance and derived value handling
How much history should move?
Enough for statement reproduction, disputes, and regulatory obligations, with the rest in an accessible archive.
Migrating everything is the default proposal and it is usually wrong: it multiplies the migration effort, inflates the target platform, and slows every rehearsal. Instead determine the obligation per data type, migrate what live processing and customer service genuinely need, and keep the remainder in an archive with a tested retrieval path and a documented retention position. The critical word is tested, because an archive nobody has queried is a liability rather than a control. Note that the archive must remain readable after the source platform is decommissioned, which frequently means extracting into an open format rather than relying on the old system.
How do you reconcile?
With arithmetic controls at several levels and zero tolerance on anything financial.
| Control | What it proves |
|---|---|
| Record counts by object and status | Nothing lost or invented |
| Control totals on key amounts | Aggregate financial integrity |
| Balance sums by product, entity, and currency | Ledger-level agreement |
| Account-level balance comparison | No offsetting errors hidden in totals |
| Hash comparison on stable fields | Field-level fidelity at scale |
| Referential integrity checks in target | Relationships survived |
| Business rule validation in target | Records are legal in the new system |
| Recomputed derived value comparison | Logic behaves equivalently |
Account-level comparison matters because aggregate totals cancel offsetting errors, which is the same reason field-level comparison beats totals in parallel running. Hold zero tolerance on financial values and handle known differences as documented exceptions with owners, not as a numeric threshold.
How do you handle in-flight items at cutover?
By inventorying every mid-lifecycle item type and deciding its treatment explicitly.
| In-flight item | Decision needed |
|---|---|
| Pending and future-dated payments | Migrate, re-establish, or complete on legacy |
| Holds and authorisations | Migrate with expiry, or release and reapply |
| Unposted transactions | Post before freeze, or migrate as pending |
| Standing instructions and direct debits | Re-establish with mandate references preserved |
| Scheduled batch jobs mid-cycle | Complete on legacy or restart in target |
| Open disputes and cases | Migrate with state, or retain on legacy with access |
| Partially disbursed facilities | Explicit balance and schedule treatment |
| Interest accrued but not applied | Migrate as accrual or apply before cutover |
This list is where conversions produce their most visible incidents, because a forgotten item type means duplicate payments or lost instructions on day one. Build the inventory from the data rather than from process documentation, by querying for records in intermediate states, and require sign-off on each treatment.
How many mock runs, and what does each prove?
Three to five full-scale rehearsals, each with a different objective.
| Run | Objective | Success looks like |
|---|---|---|
| First | Pipeline executes end to end | Completion, however many differences |
| Second | Correctness at volume | Reconciliation differences classified and falling |
| Third | Timing within the cutover window | Full run inside the available outage |
| Fourth | Full choreography including verification and sign-off | Rehearsed decision points, no improvisation |
| Fifth or dress rehearsal | Cutover as it will happen, including rollback test | Executed by the actual team, on the actual timetable |
Run at production scale rather than on a subset, since timing and failure modes are volume-dependent and a rehearsal on ten percent of data proves very little about a fixed outage window. Use the same team who will perform the real event, because the choreography is as much of the risk as the pipeline.
What does cutover day look like?
A rehearsed timetable with defined freeze, load, verify, decide, and fallback points.
Freeze the source at a stated moment with control totals captured, run the extract and pipeline, load, execute the reconciliation suite, present the evidence pack to the named decision makers, and take a documented go or no-go. Build the timetable with contingency inside the window rather than assuming everything runs at rehearsal speed, and identify the point of no return explicitly, since after it rollback means reversing live activity rather than switching a flag. Rollback must be tested rather than documented, and the criteria for invoking it should be written before the day, when nobody is under pressure. The Basel Committee's Principles for operational resilience, published in March 2021, frame this as continuity of critical services, which is the right lens for defending contingency time against schedule pressure. The wider risk framing sits in this guide to managing change risk during core-system migration.
Which metrics matter?
Reconciliation differences by control, anomaly closure, run duration against window, mapping decisions outstanding, and rehearsal count.
Report reconciliation differences per control with unexplained differences tracked separately and driven to zero. Track anomaly inventory closure, distinguishing corrected, carried forward, and excluded. Measure full-run duration against the available cutover window with the trend across rehearsals, since that number determines whether the date is achievable. Count mapping decisions still awaiting business sign-off, because those are the silent schedule risk. Report in-flight item types with agreed treatment versus outstanding. And record rehearsal count and what each proved, as that is the evidence a steering committee needs to approve a date. Delivery discipline throughout matters, and the DORA research finding that speed and stability correlate rather than trade off supports frequent full rehearsals over a single careful attempt.
Data migration is the part of a conversion where diligence is directly measurable and shortcuts are directly visible. The programmes that go smoothly are the ones that profiled early, forced every ambiguous mapping to a named owner, reconciled at account level rather than in totals, and rehearsed the whole choreography at full scale before committing to a date. The strategic framing for migrations of this kind is set out in this guide to data migration to cloud-native platforms, and the reminder that these programmes are data exercises rather than reporting exercises is made well in Solvency II as a data migration.
Frequently Asked Questions
Why does data migration decide whether a conversion succeeds?
Because the new platform can only be as correct as the data loaded into it. Logic defects surface in testing, while data defects surface as customer balances that are wrong.
Why is cleaning data later a fatal plan?
Because once records are live in the new system, corrections become customer-affecting adjustments with accounting and disclosure consequences rather than a data exercise.
What does source profiling actually reveal?
Nulls in mandatory fields, orphaned records, duplicates, out-of-range values, undocumented status codes, and encoding problems that no documentation predicted.
How much history should be migrated?
Enough to meet statement reproduction, dispute, and regulatory obligations, with the rest accessible from an archive. Migrating everything is expensive and usually unnecessary.
What reconciliation tolerance is acceptable?
Zero for anything financial. Record counts, control totals, and balance sums must match exactly, with any difference explained rather than absorbed into a threshold.
What are in-flight items and why do they cause incidents?
Pending payments, holds, unposted transactions, and scheduled instructions that exist mid-lifecycle at cutover. Each needs an explicit decision, and forgetting one produces duplicates or losses.
How many mock runs are needed?
Usually at least three to five full-scale rehearsals, each proving something different: correctness, then timing, then the full cutover choreography including rollback.
Who signs off on mapping decisions?
A named business owner per data domain, not the migration team. Every dropped field, defaulted value, and derived transformation is a business decision with consequences.
