IT services, consulting, integration for legacy systems

KISS Migration Philosophy

The Components of a Migration
In Progeni's experience there are five main components to any migration: hardware migration, program/language migration, database migration, screen/presentation migration and regression testing. The hardware issues are beyond the scope of this article. These issues have components that are unique for each platform that is being migrated to or from. Suffice it to say that it is usually the decision to migrate the hardware (e.g. mainframe to NT or Unix) that drives the decision to migrate the software. While these issues are important, they are only significant for the software migration in how they affect the choices available for the other four components. For each of the remaining components there are some issues and considerations that anyone facing a migration should study. This is followed by a discussion of the methodology that has worked very well in the many migrations Progeni has been involved with over the years.

Why Migrate?
A company will decide to migrate (as opposed to re-write or buy a package) their software because of a combination of cost savings, time savings or preservation of functionality. The cost and time savings are pretty straightforward. We have seen Java rewrite projects projected at five years or more while even large migration projects are usually less than two years. Preservation of functionality is a different matter. Over time a company's programs have encapsulated the business rules that define its business. Its software helps differentiate it from its competitors. The business rules, for the most part, MUST be preserved. If they are not preserved the unique advantage and/or the unique identity of the business could very well be lost. And because this software has evolved, the IT department, and even the business people, may not remember all the nuances of the software’s functionality in order to rewrite it.

Language Migration Issues
In most migrations of legacy systems, the language is either COBOL or a proprietary 4GL such as Mantis or System Z. There have been numerous versions of COBOL and each system manufacturer created versions of COBOL that are optimized for their platform. As a result, migrating COBOL code from one platform to another can be an adventure. Also many of today's micro based COBOL products carry runtime license fees.

Likewise 4GL languages tend to be highly proprietary. The 4GLs, while often efficient to use once you know them, are often inflexible and slow to take advantage of new technologies (such as the Web). Because many of them involve a run-time interpreter, any enhancements or change in functionality MUST come from the vendor. These run-time interpreters carry most of the enhanced functionality of the programming environment. All programs make calls to the run-time to invoke various functions. This can create a system bottleneck. Another consideration is the not inconsequential run-time license costs charged by many 4GLs.

Because of the high functionality of the source language in a typical 4GL and because the implementation of this functionality is hidden in the run-time, the applications are very difficult to migrate manually. Frequently such applications are therefore not migrated but rather rewritten with the hope that all the functionality of the old system can be duplicated. In fact I only know of one 4GL that isn’t proprietary, has NO runtime fees and affords easy migration: 1Spec from The Progeni Corporation.

Database Migration Issues
Older database technologies are often difficult to use and administer. It's also becoming increasingly difficult to find DBAs who are knowledgeable in these databases. However, because they were designed and optimized specifically for the legacy systems on which they run, these database systems are often more efficient on their native platforms than even newer relational databases that run on the same system. So the migration from one database technology to another usually goes hand-in-hand with a migration to a new technology hardware platform as well. This is not necessarily a bad thing.

Screen/Presentation Migration Issues
Migration usually involves moving away from mainframe 'green screen' terminals. Since the system I/O is most likely being moved onto a Windows or Unix based monitor with GUI or Web capabilities, the inclination is to totally redesign all the current screens to take advantage of the new presentation capabilities of the GUI or Web. However, such changes can result in the need for massive changes in the underlying software and massive retraining effort for all users. Testing the new system also becomes much more difficult because transactions don’t happen in the same way as they did previously.

Testing Issues
The most important, least talked about and poorest planned component of most migrations is regression testing. Regression testing is a specialized form of system testing. Its purpose is to test a system where any changes made to that system were not intended to affect the functionality of the system. Such changes are usually a combination of hardware platform change, program language change, database change and screen/presentation change. In other words, the environment is changing, but the functionality of the system is not. When the database has been redesigned, the screens redesigned and the reports redesigned, how do you test this new system? How do you guarantee that the new system has the same functionality as the old system? It can be done, but the job is massive. Here at Progeni, we solve this problem with a product called ThreeRs, a Regression Testing tool built specifically to solve this problem.

Other General Issues
After the migration is complete, who is going to maintain this new modernized system? This is an often-overlooked issue that is of primary concern when outside consultants are hired to perform the migration. If you migrate a 4GL, IDMS, character terminal system to a Java, Oracle, Web system, all the programmers who really know the logic and functionality of the application will need extensive retraining and are, at best, novices in the new environment. If you bring in experts on the new environment they have no clue what the application is supposed to do, let alone how it does it and what business rules it follows. If you’ve redesigned the screens and transactions your experienced users are novices in the new system as well. An additional issue that often gets overlooked is the concept we call retrofit. A manual or hand coded migration tends to take a very long time, years sometimes. During this period normal updates and maintenance are going to be necessary to the old system. How and when do you integrate or retrofit these modifications into the new system?

Methodologies
If all this scares you, it should! To quote Dale Vecchio, research director of applications development for Gartner Group, "If it was easy to get off these systems, most CIO's would have done it already.'' Migrations are hard and prone to error and don’t always achieve the results desired. According to database consultants and software developers, there are several possible ways of shifting from a legacy platform to a modern, Web-enabled application platform. They include retirement, buying an off-the-shelf software package, manual code reengineering/rewriting, cookbook rewriting, dynamic call translation, and conversion automation. Any of these methodologies can work in isolated situations, but all have their problems. The first four of these alternatives essentially throw away the existing set of business rules. The first two replace them with a "canned" set, which may or may not fit the unique personality of the business. The third and fourth replace them with the conversion programmers’ interpretation of these business rules, which may or may not be accurate. For all but the simplest most basic of applications the only method that seems to work consistently is an automated approach. Automation is the only approach that can keep the migration timeframes short enough so that the retrofit issue does not become overwhelming. Even this approach can be problematic if one additional philosophy is not also adopted.

Migration Philosophy
Over the years that Progeni has been doing migrations, we have developed a philosophy that has been instrumental in making all these migrations successful. This philosophy is simple:

When doing an environmental migration, change as little as possible to migrate the application to the new environment. When the application is running accurately in the new environment, then and only then, modify the application to start taking advantage of the benefits of the new environment.

This is a modified version of the KISS philosophy.

Most migrations fail because too much is attempted all at one time. The legacy database is converted to a relational database in third normal form. The COBOL or Mantis programs are converted to Java. The screens are migrated from character terminals to totally redesigned GUI screens for the Web. AND the application is moved from a mainframe to Unix or Windows NT. Any one of these tasks is difficult and complex. When they are all done at the same time, the complexity and difficulty grows exponentially.

Three of the main problems are:

With this totally redesigned system in a totally new environment everyone in the organization is now a novice in the new system.
There is no way to do an exhaustive regression test.
Because it takes so long to implement the new system it is very difficult to integrate ongoing required updates that were made to the old system into the new system.

The Solution
The philosophy says keep it simple, how is that accomplished? Each component of the migration has its own strategies for keeping it simple.

When migrating the screens, the old screens should map one for one to the new screens. Each of the new screens should have exactly the same fields as the old screens and each field should be in essentially the same location on the new screen. Yes, this doesn’t add all the new benefits of a full GUI Web deployment, but the retraining of the users is held to a minimum. Also, it is much easier to perform regression testing because the transactions on both systems are exactly the same. The same set of inputs run on both systems should yield exactly the same results. The "goodies" can always be added after the system is up and running in the new environment.

When migrating the database,the records/datasets on the old system should map one for one to tables in the relational database. Yes, this does not result in a fully optimized/normalized relational database, but it does require the minimum amount of restructuring of the application programs and invariably runs fast in production. This also assists the current DBA as he learns the new database technology. He may still be a novice in the new database, but the database design is what he knows, therefore he has a framework (and a full set of examples) on which to build on his knowledge and experience of the new system as he works to optimize it.

When migrating the programs, the new language should be close in design and function to the old language. This means either COBOL or a COBOL centric 4GL. The optimal 4GL will have no run-time element (and thus no runtime license fee), is extensible to include any functionality required to bring the application forward from the old language. By keeping the new programming language similar (in both syntax and functionality) to the old language the maximum amount of programmer knowledge and experience is transferred forward to the new system.

When the new system is implemented the programming staff are ready to start maintaining and enhancing the new system. Additionally retrofits are much easier. Since the program logic and structure have not changed, a retrofit is merely putting the same change into two versions of the same program. Also, it is much easier to perform regression testing because the program logic on both systems are exactly the same. The same set of inputs run on both systems should yield exactly the same results. The "goodies" can always be added after the system is up and running in the new environment.

By combining all three approaches it is very feasible to use automated tools for both the migration and regression testing. Because the program logic is not changing and the database design is not changing and the screen design is not changing, automated tools are very adept and consistent at making the syntactical changes necessary to migrate environments. Such automated tools can also document any changes that are made during the process in the form of "comments". This means that the migration is faster and less error prone than any other migration strategy. Therefore, testing is much easier and there are fewer retrofits to have to worry about. Because this is an automated syntactical migration, business rules do not get changed during the process.

Summary
The key to any successful environment migration is to keep it as simple as possible and to have a good regression test plan and methodology. Progeni has also found that the more automated the migration process, the faster and more accurate the resulting system. Once you’re on the new environment then and only then should you start to take advantage of the new technologies. The process of optimizing the system after the migration actually becomes a very good way for the legacy people (both business and IT) to "come up to speed" on the new technologies.