Continuous Delivery capability for the Multi-channel Digital Tax Platform (MDTP)

Description

Summary of the work Building and operating a Continuous Integration-Continuous Delivery (CI-CD) capability in public cloud, currently AWS, using open source technologies. Expected Contract Length 2 years Latest start date Monday 20 September 2021 Budget Range The anticipated budget is £4.8m per annum over 2 years (£9.6M total contract value). Why the Work is Being Done We are looking for a single partner with a proven track record in building and operating a Continuous Integration-Continuous Delivery (CI-CD) capability in public cloud. MDTP is a Platform-as-a-Service (PaaS) currently hosted in AWS (London Region) and all of HMRC's modern, customer-facing digital services accessed via Gov.uk reside on this platform. New digital services are constantly under development and the Platform must continue to evolve to meet the need of the teams building these services, whilst prioritising security of the platform and the services it hosts. Now in its fourth major iteration, MDTP went live in February 2014 and is always evolving to meet changing user needs. The platform vision and roadmap set out the future direction, centred on continually improving MDTP's usability, operability, security and value for money. Problem to Be Solved This is not a greenfield implementation but the continued evolution of a cutting edge, 'best of breed' PaaS. We must continue to maintain and enhance MDTP's ability to serve up online content to HMRC's customers, as well as services to internal staff, while ensuring an optimal user experience. Without a performant, secure and highly available platform, HMRC's customers will not be able to access MDTP's tenant services or may suffer a degraded user experience. The successful supplier will be required to conduct work on various areas of the project simultaneously, working in blended teams with civil servants (including apprentices and Industry Placements) and other suppliers. Who Are the Users MDTP provides a platform, with connectivity to legacy systems, for hosting the department's digital services, themselves owned and operated by the service teams based in numerous Digital Delivery Centres around the UK. MDTP's direct users are therefore the tenant service teams, with HMRC's external customers as the ultimate beneficiaries or end users. (service teams - users) As a user of MDTP, I need a performant, secure and highly available platform, So that I can deploy and operate digital services for HMRC's customers. (external customers - end users) As a user of services on tax.service.gov.uk, I need to interact with HMRC, So that I can, e.g. file my tax return. Work Already Done MDTP went live in February 2014 and is now in its fourth major iteration: 2014 - 1 x SME cloud provider 2016 - 2 x SME cloud providers (active-active) 2017 - 1 x hyperscale cloud provider (AWS) 2020 - (AWS) ECS implementation, replacing a suite of bespoke deployment orchestration tools The platform roadmap envisages MDTP's continued evolution as a cutting edge PaaS, so there is plenty of transformative feature work in addition to the live running aspects. Existing Team HMRC expects the Service Provider to work alongside other suppliers and internal staff, including apprentices, as part of blended teams. There are seven platform teams organised into technical specialisms, each of which comprises a subset of the following roles: - infrastructure engineer - software developer - QA/tester - Delivery Lead - Product Owner - Business Analyst Current Phase Live Skills & Experience • Translate business problems and user needs into technical designs • Be proficient at writing code (Scala, Python, Ruby) to solve problems, automating wherever it adds value and incorporating security best practices at all times • Have hands on experience of, as a minimum, the following AWS services - Cloudformation, S3, ECS, EC2, RDS, CloudFront, Lambda, SQS, SNS and IAM • Demonstrate proponents of test-driven development (TDD) practices, writing top quality unit tests and code • Demonstrate experience in automated testing, knowing what tests to write, providing meaningful and maintainable quality gates, ensuring features only promoted through CI-CD pipelines when they exhibit the right functional/non-functional characteristics • Have a deep understanding of distributed Source Control, preferably git/GitHub • Have a deep understanding of application deployment strategies • Have experience building automated monitoring of logs and metrics driving alerts using open source tools • Have a deep understanding of navigating and troubleshooting cloud-based Linux compute • Have experience of developing using Terraform, ELK, Grafana, Mongo • Have experience building immutable infrastructure • Have experience architecting to reduce cloud hosting costs Nice to Haves • Demonstrate a significant multi-team contract with a Public Sector body • Provide evidence of additional and/or value added activities - i.e. over and above contractual obligations - when acting in the role of a partner supplier to an organisation Work Location Bristol, 3 Glass Wharf, BS2 0PS Working Arrangments The supplier will be required to co-locate with the existing platform teams, mixed teams comprising both internal staff and other contractors. Flexibility of location is possible; however, all work must be conducted within the UK and service providers must be willing to work in Bristol whenever asked to attend. Occasional travel may be required to other HMRC Delivery Centres. Expenses will be in line with HMRC policy. It is a requirement that the successful supplier provides upskilling to permanent staff, including a growing number of junior engineers, in order to increase internal capability. Security Clearance The supplier(s) must ensure each individual supplied to us has BPSS clearance. Clearance costs to be met by supplier. Additional T&Cs All personnel engaged in the provision of this service must have a minimum of three years experience in their designated role/skill specialism, with an expectation of five years minimum experience for lead roles. Additional HMRC Mandatory Terms and Conditions will be added to the Call-Off contract. No. of Suppliers to Evaluate 4 Proposal Criteria A written proposal based on a scenario to be provided at a later date, which will be assessed against a sub-set of the Essential, Desirable and Cultural Fit criteria. Cultural Fit Criteria • Operate a no-blame culture, encouraging people to learn from their mistakes • Have excellent communication skills with staff at all levels of the organisation • Take responsibility for their work while also pairing/peer reviewing by default • Be willing to collaborate and partner, including with other suppliers and HMRC staff at all levels • Proactively share knowledge and experiences with members of team, especially with HMRC staff and in particular taking an active interest in the development of early talent • Be innovative and promote ideas and suggestions as applicable • Focus on achieving value for money in all activities, with an appreciation of the business value driving technology choices • Be able to start work immediately Payment Approach Time and materials Assessment Method Presentation Evaluation Weighting Technical competence 60% Cultural fit 20% Price 20% Questions from Suppliers 1. Currently which open source tools are being used for monitoring of logs and metrics driving alerts. The logs are stored in an ElasticSearch database, while metrics are stored in Clickhouse. There is work underway to migrate the pipeline to AWS Managed Streaming Kafka which will open other opportunities, but currently alerts are generated by Sensu querying the two datasources. 2. Please could you tell us the desired team construct, i.e. how many of each role type would be needed/comprise the current team? Currently each of teams would consist of a Product Owner, a Delivery Lead/Scrummaster and a varying number of DevOps or Developers. We do have architects and other roles but these tend to work across a number of teams. 3. Can HMRC advise if there is an incumbent supplier delivering services no and if so, who is that supplier? The incumbent supplier is eSynergy. 4. Can you please elaborate on what do you mean by immutable infrastructure? What service are you looking for? On MDTP we do not make changes to running server instances. The entire platform is defined as infrastructure-as-code and version controlled. State changes only occur when code deployments are made through the pipeline and any supplier needs to show that they understand why we choose to de-risk changes by deploying new instances rather than changing the existing infrastructure. 5. Can HMRC please provide an indicative timeline from stage 1 submission, review and selection of stage 2 candidates for proposal then award? The indicative timeline is Stage 1 Evaluation 27 July to 10 AugustStage 2 Invitation 11 August to 25 AugustStage 2 Evaluation 30 August to 10 SeptemberAward around 23 Sept to 30 Sept with an immediate start date. All dates are indicative and HRMC reserve the right to change as needed so suppliers must be available to start from mid September. 6. Can HMRC provide a timeline to the stage 2 evaluation of suppliers, is there a timeline for award and an expected start date for the winning bidder? The indicative timeline is Stage 1 Evaluation 27 July to 10 AugustStage 2 Invitation 11 August to 25 AugustStage 2 Evaluation 30 August to 10 SeptemberAward around 23 Sept to 30 Sept with an immediate start date. All dates are indicative and we reserve the right to change as needed so suppliers must be available to start from mid September. 7. For experience and skills can we reference the experience of individuals or are you seeking our company project based experience ? Since the contract will be awarded to a company, the experience of your company would be paramount. That being said, if the experience of individuals is applicable and can be considered indicative of the overall companies' experience we would not consider there to necessarily be a distinction. 8. We note the Authority’s requirement concerning co-location. Can the Authority confirm how many (or what percentage of) resources it expects to be based in Bristol? MDTP at this current time is almost entirely working remotely, with most of our engineers working in or near Bristol. Whilst this may not change completely in the future we do ask that people are based in Bristol or at least can get to Bristol to take part in team sessions that work better in person as these can be ad-hoc. 9. For the support component, what are the Authority’s expectations concerning availability? (i.e., standard business hours, on-call, or 24/7?) MDTP is supported 24x7. There is an Out of Hours support rota, with a single engineer on call and, given the platform's stability, it is more often the case that no action is required overnight. Each platform team also has an 'in hours' support rota where they are expected to provide support to our users. There are typically low volumes of support tickets and so the engineer may return to pairing on other work in the backlog. 10. Can the Authority confirm, in either exact or percentage terms, the split between ‘business as usual’ and ‘feature development’ items on its backlog? No, I don't believe that is something we are able to quantify in any exact measure. With BAU tasks we would look to automate them - but would likely consider the development of that automation to be a new feature. We are a mature platform and most work could be considered iterative rather than greenfield in nature. Our Statements of work focus on discrete pieces of work, but each team will have BAU tasks that would also be necessary for each period. 11. Can HMRC please confirm that there is no harm in providing more than one piece of evidence in a single question? We are not prescriptive as to whether suppliers offer a single piece of evidence or multiple pieces. The most important thing to keep in mind is how comprehensively the evidence addresses the criteria. 12. What deployment strategies are used for MDTP platform? Currently the only deployment strategy that we make available is a rolling deployment. This ensures zero downtime during a deployment of a new version of a service. There is no reason why we couldn't offer different deployment strategies in the future, but it is not currently supported. 13. In terms of building and operating the CI/CD capability in AWS, would the supplier be responsible for all pipeline stages? Because of the way the platform teams are organised into technical specialisms, each of which is a blended team, this is unlikely. 14. Does the platform still consist of 130 user-facing applications and 900 micro-services? As of June 2021 there were 244 customer-facing applications and around 850 Production microservices. 15. What is the role of Made Tech and Equal Experts in this platform? Madetech and Equal Experts, along with the supplier awarded this contract, are delivery partners who work together in blended teams alongside HMRC's permanent staff. 16. Are there any uses of EKS, CodeBuild, CodeCommit, CodeDeploy or CodePipeline? There is some use of CodeBuild by platform teams but it is not provided as a service for our users. MDTP utilises ECS and not EKS. 17. Has AWS CloudWatch been ruled out for the purposes of monitoring? AWS CloudWatch is used by our Platform Teams, however it is not currently used by teams hosting services on MDTP. The reason for this is largely because we have had Sensu in place for a long time. However any monitoring mechanism used by the service teams would need to be configurable as self service by those service teams. Since service teams do not have AWS credentials that would add a degree of complexity. 18. What is currently used for static and security analysis? OWASP ZAP has been implemented in the build pipeline for dynamic security testing. Penetration testing is a requirement for new services joining the platform, and annually or at other specified times thereafter. Aside from technologies we use techniques such as threat modelling and conduct regular security reviews. 19. Does this contract include the management of all dependent services (Internet gateways, NAT gateways, application load balancers, Route 53 service registries, etc.)? Yes, everything inside the MDTP VPCs, including connectivity to the Internet and HMRC's integration layer/back end. Since MDTP comprises seven platform teams segmented into technical or functional specialisms, the actual work packages to be delivered will depend on which platform team(s) the successful supplier is onboarded to. 20. Can the Authority confirm to what extent, if any at all, the scores from the first stage will be carried through/weighted to the second stage? Stage 1 scores will be carried through for the shortlisted suppliers and added to stage 2 scores forming the total Technical Competence score which carries a weighting of 60% 21. Can the Authority share with us the high-level roadmap for its platform? Due to technical difficulties we are unable to share one here however one will be shared at stage 2 with the shortlisted suppliers.