With the Scottish Government about to embark on a Cloud First policy to migrate their estate of legacy data centres and applications to the Cloud, a critical skill set needed for the local market is ‘Agile Cloud Migration’ – Extending the scope of Agile development teams to include the migration of legacy applications.
Building new Cloud services from scratch is a relatively simple proposition and is why startups can take to Cloud adoption quickly and easily.
For large enterprise organizations like government their primary challenge is their existing legacy systems, they operate huge data centres that over time have accumulated multiple eras of hardware platforms, operating systems, databases and applications, stretching as far back as the original mainframes.
There is also a long tail of thousands of Microsoft Access databases and Excel spreadsheets littered across departmental PCs, and broader still, the use of paper-based forms that must be completed by hand to execute some processes, or at best downloaded and submitted via email.
In many cases the original skills required to maintain these systems are long gone and the systems have hardened to become black boxes, the organization dare not try and adapt them lest they break and no one knows how to fix them, but they still work and so are kept in place.
For the Government sector elderly systems like COBOL are still prevalent, indeed in the USA they account for 70% of IT spend, and cost the government nearly $40 billion a year to maintain. This is why they aren’t investing as much in new innovation-enabling technologies like Cloud as they might.
In Oct 15 the UK Authority web site reported that the National Audit Office said the public sector is still struggling to master and realize the potential of digital transformation, despite the citizen and cost benefits it’s known to deliver.
They also identified legacy applications as the root cause of this lack of progress in all of these areas, reporting that over £480 billion of government revenues were reliant on them highlighting the many risks this presents, most notably resistance to the new digital innovations governments are required to adopt to achieve new online services:
“The government’s ICT strategy, published in March 2011, recognized legacy ICT as a barrier to the rapid introduction of new policies and particularly the move to ‘digital by default’.
Legacy ICT reduces the flexibility to improve public services, makes it harder to protect against evolving cyber threats and increases government’s reliance on long-term contracts with large ICT companies. It is also likely to increase the cost of operating public services by preventing higher levels of automation and hinder data sharing intended to prevent fraud and error.”
In their audit they review a sample of government department situations and their legacy application challenges – the DWP Pension Service, HMRC VAT Collection, NHS Prescription Payment Service and the OFTs Consumer Credit Licencing Service.
These scenarios feature a variety of aged technologies, some originating as far back as 1973 running on a mainframe computer. The HMRC identified in 2009 that their 600 systems were “complex, ageing and costly”, and the report highlights how expensive a burden this is: The VAT collection service costs £430 million per annum to operate, and the DWP’s Pension Payment service £385 million per annum. That’s almost a billion pounds a year just for two applications.
Simply ‘lifting and shifting‘ these apps to the Cloud, ie virtualizing and deploying them to IaaS, won’t address the bulk of this challenge. Yes it will tackle the issue of aged hardware but the code remains as is.
The critical consequence of this is that the code is therefore still as difficult to modify as before, no transformational benefits have been achieved, it is still an inhibitor to digital transformation.
Therefore the first aspect and defining goal of Agile Cloud Migration is to migrate these systems into new DevOps environments so that this becomes possible, and new innovations can be developed and deployed at the fast pace these techniques make possible.
Legacy modernization best practices can address these issues, delivering business benefits including:
- Untangle and map legacy application complexities – Build a basis of understanding of existing application and data architectures to establish more intelligent IT planning concepts in line with business and technical demands. Developers with no experience of the legacy software can be enabled to implement changes in line with business needs.
- Extend the life of legacy applications without the risks of greenfield COTS projects – Numerous reports highlight how a COTS (Commercial Off The Shelf) approach to modernization is very high risk with expensive failure rates.
- Align user interfaces and back-end application and data models with modern business processes – Modernization can be used to achieve IT objectives such as SOA, Cloud migration and Web-enablement of applications.
- Leverage new technologies and tools – The overarching benefit is the transformation of software that is now resistant to change and thus innovation, as the required skills have long since retired and/or the suppliers are no longer in business. By moving it to a modern software platform new tools and techniques like ‘DevOps’ can be implemented to speed the rates of innovation.
A couple of case studies illustrate the basic principles for AWS and Azure respectively:
Netflix – Netflix is the poster child for ‘Cloud Native’ development but at one time they too operated a traditional enterprise IT approach. To achieve the massively disruptive digital services platform they now operate they underwent a holistic migration and transformation, to the AWS Cloud.
In this blog they focus on the migration of the core Netflix billing systems from their own data centre to AWS, and from Oracle to a Cassandra / MySQL combination, emphasizing in particular the scale and complexity of this database migration part of the Cloud Migration journey.
They also reference a previous blog also describing this overall AWS journey, again quickly making the most incisive point – this time describing the primary inflection point in CIO decision making that this shift represents, a move to ‘Web Scale IT‘:
That is when we realized that we had to move away from vertically scaled single points of failure, like relational databases in our datacenter, towards highly reliable, horizontally scalable, distributed systems in the cloud.
Microsoft MS Sales – Microsoft’s core Revenue Reporting system had reached the limits of both the underlying infrastructure and also the application functionality essential to agile competitiveness.
It was identified that Lift and Shift only would not address the latter challenge, and so additionally the apps were modernized for Azure PaaS to leverage a Microservices and Big Data architecture.
Monolith to Microservices
The Microsoft case study highlights the essential dimension to this transformative Cloud migration approach – Modernizing the core architecture of the enterprise software, from a monolith to a microservices model.
A microservices software architecture is the pinnacle of Cloud Native computing, and is relatively simple to understand when considering greenfield projects, but for most enterprise organizations it quickly brings them back around to the topic of legacy modernization, requiring a much more complex challenge of how to adapt their existing systems to this new approach. InfoQ offers a great series of articles on the topic. That poor old monolith, you can migrate it, transform it, decompose it, break it, smash it, or just skip it.
This presentation from Linkedin offers a detailed case study, describing their approach for exactly this scenario – From a Monolith to Microservices + REST:
- A legacy estate of Java, Servlets, JSP and Oracle databases.
- A need to support fast release iterations as far back as 2010, which ran into the core challenges associated with monolith software: Test failures, rollback difficulties and complex orchestration and dependencies between services.
- So they broke apart the codebase, adopted Continuous Delivery practices and devolved controls, implementing a decentralized code base.
- The use of Java RPC meant a proliferation of APIs made backwards compatibility a big problem, a situation they addressed by moving to Rest.li , a REST + JSON framework, key components from the Netflix suite – Apache Zookeeper for dynamic service discovery, and DECO for URN resolution to explore data graphs.
This combination formed their particular ‘Microservices Recipe’, and when you consider the role social graphs play across the Linkedin environment, how our business contacts are inter-connected and we dynamically explore our way through them, you can see how it would be an ideal design for this type of web site.
Others offer very practical permutations. For example in this article Flickr describe how you can utilize Github to operate a ‘Microservices Store’.
“Some of the products that we work with at Yahoo have a very granular architecture with hundreds of micro-services working together. For scenarios like this, it’s convenient to store configurations for all services in a single repository. It greatly reduces the overhead of maintaining multiple repositories. We support this use case by having multiple top-level directories, each holding configurations for one service only.”
This is a great idea when you consider Github can provide the foundation for a complete DevOps toolchain, augmented in many ways such as adding apps to support Agile practices.
Similarly Sensedia propose a recipe for Legacy Modernization that defines how microservices can be utilized as an API enablement strategy.
Chandra Rajasekharaiah, Enterprise Solutions Architect at Macy’s, published this excellent deep dive analysis of the Monolith to Microservices transformation and the software engineering challenges it presents, and Anil Madan, VP of Engineering at Intuit also describes the same journey encompassing a broader perspective of platforms and organizations.
Agile Cloud Migration
AWS offers a wealth of insights developed from their experience of having now migrated hundreds of enterprise customers to their Cloud.
For example this presentation describes an ‘Agile Approach to Mass Migrations‘, providing a comprehensive primer on a wholesale transformation framework that is based on and can integrate with existing Agile practices, achieved through building a Cloud Centre of Excellence and referencing thought leaders such as Gene Kim’s Phoenix Project and Jez Humble’s Lean Enterprise principles.
Best practice resources
They also offer an extensive supporting library of further resources:
- Their Migration Hub brings these practices together into a knowledge base.
- A Cloud Transformation Maturity Model for assessment and planning, supported by presentations including an AWS Migration Planning and Roadmap, a guide for Migrating Enterprise Applications to AWS and Application Migrations at Scale, and Governace Strategies for Cloud Transformation.