M&M Data Migration

M&M — short for Migration von Massendaten, migration of mass data — is a batch system that moved roughly 20 million person records and more than 50 million accounts off a 1970s IBM mainframe and into the object model of a new Java application. It ran in production exactly once.

That “exactly once” is what makes it worth reading. A system with a single production run has no modifiability requirements, no security requirements worth the name, and no user interface — so almost everything in this architecture is driven by just two goals: get through 20 million people in under 24 hours, and be able to prove afterwards that every record was handled correctly. The solution strategy is three lines long, and you can trace nearly every decision back to one of those two goals.

It is also a good example of a documentation that says out loud where it stops. Several building blocks are deliberately left as blackboxes, the quality scenarios and the risk section are marked as omitted, and the author notes at each point what a real documentation would have to add instead.

Notes on the System

A few words of clarification: this project really happened, and the M&M system really was built. The client, “Fies und Teuer AG”, was of course called something else in reality, and the business content has been simplified a little.

The real migration project took place around 2002/2003 somewhere in Germany. Over a good twelve months, more than ten domain experts analysed the logic and business rules of the existing mainframe systems and built the business migration concept on that basis. This had to take account of the particular requirements for proof and record-keeping, and of the legal conditions that had changed over time. Interlocked with that work, a team of more than ten developers and architects built the matching software.

Some of the stumbling blocks and challenges of the “real” M&M project are collected in section 11.