In this earlier blog I introduce and suggest a new technology trend of ‘Self-Sovereign Identity’ is a very powerful development, and that there’s likely an important connection with a nations ambition to achieve self-sovereignty, as you might imagine.
An especially helpful primer to this technology is offered through this webinar, which describes it from a practical, day to day value point of view, in terms of how it might be adopted by governments for functions like online drivers licence services.
John Jordon, Province of British Columbia, describing their work around Digital Identity, collaborating with the Federal and Ontario governments too. He lays out how they have been and still are today globally pioneering the role identity plays in facilitating Digital Government services.
My favourite part is the introduction, where John describes how the part of Canada he comes from is where the beavers and cows are. That’s just peak Canadian.
With the Scottish Government about to embark on a Cloud First policy to migrate their estate of legacy data centres and applications to the Cloud, a critical skill set needed for the local market is ‘Agile Cloud Migration’ – Extending the scope of Agile development teams to include the migration of legacy applications.
Building new Cloud services from scratch is a relatively simple proposition and is why startups can take to Cloud adoption quickly and easily.
For large enterprise organizations like government their primary challenge is their existing legacy systems, they operate huge data centres that over time have accumulated multiple eras of hardware platforms, operating systems, databases and applications, stretching as far back as the original mainframes.
There is also a long tail of thousands of Microsoft Access databases and Excel spreadsheets littered across departmental PCs, and broader still, the use of paper-based forms that must be completed by hand to execute some processes, or at best downloaded and submitted via email.
In many cases the original skills required to maintain these systems are long gone and the systems have hardened to become black boxes, the organization dare not try and adapt them lest they break and no one knows how to fix them, but they still work and so are kept in place.
In Oct 15 the UK Authority web site reported that the National Audit Office said the public sector is still struggling to master and realize the potential of digital transformation, despite the citizen and cost benefits it’s known to deliver.
They also identified legacy applications as the root cause of this lack of progress in all of these areas, reporting that over £480 billion of government revenues were reliant on them highlighting the many risks this presents, most notably resistance to the new digital innovations governments are required to adopt to achieve new online services:
“The government’s ICT strategy, published in March 2011, recognized legacy ICT as a barrier to the rapid introduction of new policies and particularly the move to ‘digital by default’.
Legacy ICT reduces the flexibility to improve public services, makes it harder to protect against evolving cyber threats and increases government’s reliance on long-term contracts with large ICT companies. It is also likely to increase the cost of operating public services by preventing higher levels of automation and hinder data sharing intended to prevent fraud and error.”
In their audit they review a sample of government department situations and their legacy application challenges – the DWP Pension Service, HMRC VAT Collection, NHS Prescription Payment Service and the OFTs Consumer Credit Licencing Service.
These scenarios feature a variety of aged technologies, some originating as far back as 1973 running on a mainframe computer. The HMRC identified in 2009 that their 600 systems were “complex, ageing and costly”, and the report highlights how expensive a burden this is: The VAT collection service costs £430 million per annum to operate, and the DWP’s Pension Payment service £385 million per annum. That’s almost a billion pounds a year just for two applications.
Simply ‘lifting and shifting‘ these apps to the Cloud, ie virtualizing and deploying them to IaaS, won’t address the bulk of this challenge. Yes it will tackle the issue of aged hardware but the code remains as is.
The critical consequence of this is that the code is therefore still as difficult to modify as before, no transformational benefits have been achieved, it is still an inhibitor to digital transformation.
Therefore the first aspect and defining goal of Agile Cloud Migration is to migrate these systems into new DevOps environments so that this becomes possible, and new innovations can be developed and deployed at the fast pace these techniques make possible.
Legacy modernization best practices can address these issues, delivering business benefits including:
Untangle and map legacy application complexities – Build a basis of understanding of existing application and data architectures to establish more intelligent IT planning concepts in line with business and technical demands. Developers with no experience of the legacy software can be enabled to implement changes in line with business needs.
Extend the life of legacy applications without the risks of greenfield COTS projects – Numerous reports highlight how a COTS (Commercial Off The Shelf) approach to modernization is very high risk with expensive failure rates.
Align user interfaces and back-end application and data models with modern business processes – Modernization can be used to achieve IT objectives such as SOA, Cloud migration and Web-enablement of applications.
Leverage new technologies and tools – The overarching benefit is the transformation of software that is now resistant to change and thus innovation, as the required skills have long since retired and/or the suppliers are no longer in business. By moving it to a modern software platform new tools and techniques like ‘DevOps’ can be implemented to speed the rates of innovation.
A couple of case studies illustrate the basic principles for AWS and Azure respectively:
Netflix – Netflix is the poster child for ‘Cloud Native’ development but at one time they too operated a traditional enterprise IT approach. To achieve the massively disruptive digital services platform they now operate they underwent a holistic migration and transformation, to the AWS Cloud.
In this blog they focus on the migration of the core Netflix billing systems from their own data centre to AWS, and from Oracle to a Cassandra / MySQL combination, emphasizing in particular the scale and complexity of this database migration part of the Cloud Migration journey.
They also reference a previous blog also describing this overall AWS journey, again quickly making the most incisive point – this time describing the primary inflection point in CIO decision making that this shift represents, a move to ‘Web Scale IT‘:
That is when we realized that we had to move away from vertically scaled single points of failure, like relational databases in our datacenter, towards highly reliable, horizontally scalable, distributed systems in the cloud.
Microsoft MS Sales – Microsoft’s core Revenue Reporting system had reached the limits of both the underlying infrastructure and also the application functionality essential to agile competitiveness.
It was identified that Lift and Shift only would not address the latter challenge, and so additionally the apps were modernized for Azure PaaS to leverage a Microservices and Big Data architecture.
Monolith to Microservices
The Microsoft case study highlights the essential dimension to this transformative Cloud migration approach – Modernizing the core architecture of the enterprise software, from a monolith to a microservices model.
A microservices software architecture is the pinnacle of Cloud Native computing, and is relatively simple to understand when considering greenfield projects, but for most enterprise organizations it quickly brings them back around to the topic of legacy modernization, requiring a much more complex challenge of how to adapt their existing systems to this new approach. InfoQ offers a great series of articles on the topic. That poor old monolith, you can migrate it, transform it, decompose it, break it, smash it, or just skip it.
A legacy estate of Java, Servlets, JSP and Oracle databases.
A need to support fast release iterations as far back as 2010, which ran into the core challenges associated with monolith software: Test failures, rollback difficulties and complex orchestration and dependencies between services.
So they broke apart the codebase, adopted Continuous Delivery practices and devolved controls, implementing a decentralized code base.
The use of Java RPC meant a proliferation of APIs made backwards compatibility a big problem, a situation they addressed by moving to Rest.li , a REST + JSON framework, key components from the Netflix suite – Apache Zookeeper for dynamic service discovery, and DECO for URN resolution to explore data graphs.
This combination formed their particular ‘Microservices Recipe’, and when you consider the role social graphs play across the Linkedin environment, how our business contacts are inter-connected and we dynamically explore our way through them, you can see how it would be an ideal design for this type of web site.
Others offer very practical permutations. For example in this article Flickr describe how you can utilize Github to operate a ‘Microservices Store’.
“Some of the products that we work with at Yahoo have a very granular architecture with hundreds of micro-services working together. For scenarios like this, it’s convenient to store configurations for all services in a single repository. It greatly reduces the overhead of maintaining multiple repositories. We support this use case by having multiple top-level directories, each holding configurations for one service only.”
This is a great idea when you consider Github can provide the foundation for a complete DevOps toolchain, augmented in many ways such as adding apps to support Agile practices.
Similarly Sensedia propose a recipe for Legacy Modernization that defines how microservices can be utilized as an API enablement strategy.
Chandra Rajasekharaiah, Enterprise Solutions Architect at Macy’s, published this excellent deep dive analysis of the Monolith to Microservices transformation and the software engineering challenges it presents, and Anil Madan, VP of Engineering at Intuit also describes the same journey encompassing a broader perspective of platforms and organizations.
Agile Cloud Migration
AWS offers a wealth of insights developed from their experience of having now migrated hundreds of enterprise customers to their Cloud.
For example this presentation describes an ‘Agile Approach to Mass Migrations‘, providing a comprehensive primer on a wholesale transformation framework that is based on and can integrate with existing Agile practices, achieved through building a Cloud Centre of Excellence and referencing thought leaders such as Gene Kim’s Phoenix Project and Jez Humble’s Lean Enterprise principles.
Best practice resources
They also offer an extensive supporting library of further resources:
Their Migration Hub brings these practices together into a knowledge base.
Mandate the use of common platforms and infrastructure, including cloud hosting, as appropriate across the Scottish Government.
Make better use of cloud-based solutions as a source of both cost reduction and service innovation, and move public sector data hosting to a cloud environment wherever this is appropriate in terms of security and efficiency.
Niall Creech, Head of Cloud Engineering at the Ministry of Justice, provides this excellent case study of the type of benefits Scottish agencies can seek from doing so, documenting their move to the AWS Cloud as shifting to Government at Scale.
Other agencies like the Home Office are also seeking to replicate the move.
Niall makes the point that in today’s IT world there is little value to be had in operating traditional data centres any more, and moving to the Cloud represents “moving out of the basement”.
It’s not simply a process of outsourcing, transfering the same technology paradign from in-house to an external supplier, but of harnessing an entirely new paradigm all together.
The case study offers a very articulate definition of how Cloud provides an ‘Agile Infrastructure’. Instead of just migrating the same virtual servers to IaaS, Niall describes how they have embraced Cloud Native building blocks, such as containerized applications, serverless functions and elastic storage, to make possible more dynamic and agile provisioning and management of IT infrastructure. For example all live services have the ability to have any of their servers destroyed without notice, with no alerts and no user impact.
By creating apps through composing together AWS services as building blocks, and automating deployment through nested stack templates the team are able to abstract themselves away from low level administrative work, enabling a focus on value generating digital services, not IT operations:
We were also quick to understand the value of freeing people from maintenance, data security, and disaster recovery that key managed services like Amazon RDS gives.
Even with high degrees of automation, maintaining a growing infrastructure places a burden on an organisation that can hold it back from achieving its core aim, developing and providing people with the digital services they need from modern government.
Other case studies include Derby City Council, who have similarly moved out of the basement to drive cost reduction of on-premise software licencing, among other benefits including:
achieved significant reductions in total ICT spend
improved server monitoring and optimisation
the ability to respond more flexibly to changing business needs
transferred repetitive tasks, such as server patching, to a supplier
reduced the time, effort and cost to procure and manage new services
accessed suppliers with niche skills that traditional outsourcing would exclude
Similarly the DWP saved £20m a year by migrating their ‘Tell Us Once’ digital service away from a single, non-extendable managed contract. This case study also highlights the agile nature of the G-Cloud procurement process too; under very tight timescales they were able to utilize the fast-moving process to secure the deal in only 8 weeks.
Nottinghamshire Fire and Rescue Service faced a challenge of a complex legacy telephony equipment causing excessive costs and becoming entirely obsolete with no support. Again leveraging the G-Cloud procurement marketplace they sourced a modern Skype for Business based solution, one that provided all of the same core features they needed plus a host of new ones, such as multimedia collaboration, intelligent call routing and integration with fire station PA systems to announce emergency messages.
The UK Government’s Land Registry documented their digital strategy plan here; this can form a general digital transformation template model for any government agencies, and also provides specific innovation insights for other land registry organizations.
The first simple component of the open government policy template in action is the first paragraph defining the document itself as available via an open source licence.
Brilliant at the Basics: Case Management Digitization %
It’s an ideal baseline for a maturity model building reference case study, particularly as their headline theme of ‘Brilliant at the Basics’ is ideal for characterizing of the first step on the ladder.
The Land Registry case workers process 20,000 applications per day, with 650,000 received via post annually. Via a set of ‘Customer Targets’ they specify KPIs for the response rate levels expected of them for handling these, as well as ultimately a customer satisfaction rating for doing so.
The basic building block of Digital Government is the digitization of core business workflows like these claims, and the Registry reported an 81% level of automation in 2016, with a target of achieving 95% by 2022.
Having detailed Case Management targets like the Land Registry is thus the ideal end state of a basic level of digital maturity.
Similarly they also state ‘Digital Targets’ for their performance goals in a second main area of focus of digital innovations – What might they offer in the future that they don’t currently, and how might this benefits their customers and ‘disrupt’ their sector?
Digital Targets build on the previous KPIs, such as defining a metric of ‘we will average at least 99.6% availability for external e-services during published service hours’.
It also identifies specific technological advances and the new digital services they might make possible, for example launching ‘the Digital Mortgage service to create, sign and register a mortgage for approved partners‘.
A key component of this focus is the inclusion of specific key technology trends, most notably Open Data and the Blockchain. Highlighting that:
HM Land Registry holds a rich variety of publishable data of potential interest to individuals, customers, governments, lawyers, conveyancers and financial institutions. Every plot of registered land has a unique register title with corresponding number and plan. Information stored relates to people, places, rights and restrictions.
the Land Registry describes how they publish a number of open data sets and highlights the essential dynamic, how these enable new digital services, such as MapSearch.
This all culminates towards their pinnacle digital initiative ‘Digital Street’, “groundbreaking research that is exploring how land registration might work in 2030”, and is also how they express their highest stretch target of pioneering a world-class capability:
We will then apply real-life scenarios to demonstrate how a fully geospatially enabled digital register might revolutionise future property transactions, including using Blockchain technology, putting HM Land Registry at the forefront of global land registration innovation.
Digital Street is a new prototype registry that will enable this new real-time, machine-readable information platform.
The goals are to harness AI and other emergent tech and apply them towards impacts like improving the conveyancing process. Here is where they see the potential for Blockchain in particular:
New technologies such as Blockchain might enable the register to be distributed among trusted parties such as lenders and conveyancers, giving them the ability to operate and update in a secure and tamper-proof manner. We will explore the benefits of this alternative approach to managing a digital land register.
Digital Business Model Canvas
Their digital plan also outlines how to build a ‘digital business model canvas’ – Before web sites and code the team should have a clear understanding of the who, what and how of the new digital services.
This is an ideal early ‘forming’ process of bringing a team together, and can be populated from landscape reviews of what other similar organizations are doing. For example comparable initiatives include the Future Cities Catapult Land Information Platform, and in Estonia E-Estonia e-land register.
A key section to review is on page 14 where they describe their approach to exploring and formulating a customer/partner segmentation framework for expressing these as ‘digital ecosystems’, such as clustering groups like FinTech, LawTech and PropTech. Jargonistic yes but actually very helpful for populating a business model canvas and planning related activities.
The primary ‘value chain’ built atop the Land Registry is naturally all related processes: Conveyancing, property building, property selling, .. etc. This implements particular workflows across those defined groups and there is no doubt there is still much opportunity for better digital services between them. Digital business model canvas planning is the ideal way to draw out these service ideas.
Identity-Enabled Digital Services
What’s immediately notable is the involvement of Identity technologies too, the two are practically inseparable.
For example this enabling role of new digital service innovations is effectively highlighted in the UK Government’s HM Land Registry digital business plan, in which they describe a future intention of:
“We will continue to enhance and expand our existing e-enabled services, such as our innovative Digital Mortgage service that uses secure identity assurance techniques to enable a customer to sign their mortgage deed digitally. Across all stages of digital service development, the needs of the user and a drive for speed and simplicity will be at the heart of our design.
we will launch a Developer service to provide a digital online lodgement and approval of estate plans and template leases and transfers.
we will launch a Digital Transfer service to create, sign and register a transfer of a registered title.”
Business Secretary Greg Clark believes these Digital Mortgages could help reduce fraud.
The role of Identity highlights services / organizations such as Gov.UK Verify. For example in this PropertySolvers article the author explores the scenario, in particular noting the HM Land Registry ambitions:
“Graham Farrant, chief executive and chief land registrar at the HMLR reported on testing the Sign Your Mortgage Deed service for remortgaging homeowners.
This is especially notable because it is a significant technological leap, to accept Identity-related authorizations from a third-party rather than directly yourself, and this concept is currently experiencing teething issues, with Verify receiving considerable negative press for adoption and user abandonment issues.
For example the Law Gazette writes specifically on this relevant to this evolution of e-conveyancing, highlighting the basic evolution to online service is essential but the choice of Verify adds an element of doubt.
New technology innovations provide a further stimulus through introducing new capabilities that weren’t previously possible. Where Open Data makes information more accessible and programmable, the Blockchain provides a more secure transaction platform.
Given the Blockchain is fundamentally a registry system then naturally all registry-related activities are a fertile area for its application.
As Anglia Research describes it is specifically intended as a ledger system for enabling and recording asset transfers, and CoinDesk provides this detailed examination of the use case, highlighting how one aspect of natural disasters is the potential loss of all paper based versions of these records, avoidable through encoding them on the Blockchain.
Scotland operates their own registry organization through ‘RoS’ – Registers of Scotland.
At 400 years old the Registers of Scotland is the world’s oldest public land register, but today act as a beacon of excellence in harnessing technology modernization towards this goal, for all of Scotland’s public sector to follow.
This includes better utlizing their data to personalize online services, like producing bespoke reports for clients such as the Royal Bank of Scotland case study, which describes how they provide weekly reports to eliminate the considerable manual workfload the bank mortgage team faced when monitoring their applications.
The registers are central to a range of industry workflows, for example the Development Plan Approval is used for new housing developments, and they have improved this process by working collaboratively with key users like house builders and solicitors, as described in video interviews with Taylor Wimpey and Gillespie MacAndrew.
Their blog provides a peek Inside RoS, and a web library in the style of Gov.uk offering literature such as new deed plans.
Open Data Mapping
Their strategies are more or less identical including the focus on open data mapping as the flagship digital initiative, the launch of ScotLIS bringing a similar capability to market as Digital Street.
Open data mapping and blockchain technologies are each alone hugely powerful technologies to exploit, and with such a natural synergy given the role of ‘trusted ledgers’ and registry organization, when combined through this scenario will prove especially
There is considerable potential for other related agencies too. For example the delightful juxtapose of such ancient documents being transposed to such modern digital facilites, like the St Giles Cathedral, also highlight how this content would be ideal for ‘digital tourism’ too – Showcasing this heritage to the broader world to attract more online attention to Scotland and ultimately visitors too.