Enhancing Hackney’s Data Platform so that it is so good analysts want to use it

Description

Summary of the work Hackney is developing a cloud-based data platform to enable deeper insight, break down data silos, put security at the heart of our approach, and promote stability through reusable, scalable processes. We need to enhance the platform and increase adoption from analysts across the council. Expected Contract Length 6 months Latest start date Monday 10 January 2022 Budget Range We expect to spend up to £293,000 + VAT on this phase of work. We will collaboratively agree smaller phases through Statements of Work to ensure we continue to meet user needs. Why the Work is Being Done Hackney manages personally identifiable and sensitive data about its residents. This rich range of data can generate valuable insights about our people, their needs and the impact of our services. In order to derive insight from our data, we are building a cloud data platform to: - Put security at the heart of our approach: Data must be stored, transferred, and accessed securely at every stage so that residents are assured we are handling their data responsibly and we meet our legal obligations. We need joined up information governance processes so that users can easily request and grant access to data. - Break down technology and service data silos: We need to enable officers across the council to understand what data is available; minimise and anonymise data so that it can be shared more easily; and enable analysts to use data in a range of analytical tools that meet their needs. - Embed reusable, scalable processes: Dependable, reusable, automated processes will save time and allow analysts to focus on higher value tasks. Analysts also need to see, learn from, and reuse each other’s work to reduce duplication and generate meaningful insights. Problem to Be Solved We have been developing our data platform since March 2021 and put key processes in place to ingest, transform, and publish data. We now need increase adoption so that analysts can use the platform in their day-to-day work through a holistic approach that considers: - Skills & Capabilities: the platform will introduce new tools and ways of working for analysts, so they need access to learning resources and support in order to use the platform effectively. - Culture & Mindset: previously many services relied on ICT to get access to their own data. Our approach gives analysts more visibility and ownership of their data pipelines. - Products & Services: we need to ensure that our playbook enables users to share and reuse each other’s work, and our catalogue enables users to find and understand data. - Technology & Data: we need to ensure that we have the right tools and processes in place so that users can ingest, transform and analyse data. - Systems & Processes: we need to develop our approach to onboarding users and supporting them to use the platform. We also need to join up the platform with our information governance processes. Who Are the Users At a very high level, the user needs for our data platform are: - As a resident, I need my data to be managed securely and responsibly so that my personal information is secure and I can trust the council to deliver a fair and transparent service to meet my needs. - As a service manager, I need timely insights about the effectiveness of my service so that I can take action and improve outcomes for residents. - As a data analyst, I need to access, manipulate and analyse a range of data sources through the data platform so that I can generate meaningful insights more quickly and robustly. - As an engineer, I need to create reusable tools and processes in the data platform which require minimal maintenance so that I can focus on developing and improving the platform. Further detail about the profile of our users, more detailed user needs, and prioritisation is set out in this document (available on request): https://docs.google.com/presentation/d/1Bp9hUPKnMnNQDM8I2S-Jh2kdwF4xx2zvKyf-GLn61ZM/edit?usp=sharing Work Already Done The first phase of developing our data platform was done in partnership with MadeTech. The following documents provide a summary of the work to date: - A summary slidedeck of our objectives, work to date, and current position (available on request): https://docs.google.com/presentation/d/1s1vJcUMZOYsSSuYc7B_c2O7AzJwps487dePGQjXh4Tw/edit?usp=sharing - Our project weeknotes (available on request): https://docs.google.com/document/d/1ppU4xCmpShHYDDDPHjywzTmnJq-sfjgH4Iewi6dfQJw/edit?usp=sharing - The Data Platform Playbook which provides documentation of our existing processes and architectural decisions: https://playbook.hackney.gov.uk/Data-Platform-Playbook/ Existing Team Hackney would expect to provide: - A product manager - 2-4 data analysts on a part time basis - A junior developer - A delivery manager - A data engineer - User research capability As part of this work we are aiming to recruit 3 in-house developers. We also have an emerging community of practice of around 50 analysts across the council. The successful agency will bring particular value through: - Experienced developers with experience of the toolset - Experienced data engineering - Experienced data architect who can advise on the design of the platform Current Phase Beta Skills & Experience • Demonstrate an understanding of some of the key services, datasets and analytical capacity within a local authority • Demonstrate a track record of user-centred design and Agile approaches to delivery • Demonstrate a track record of mentoring junior developers and teams to become self-sufficient • Experience of solution architecture within a broader organisational framework to achieve security, resilience and value for money • Experience of using AWS S3, Glue, and Redshift to deliver data pipelines • Experience of using Python and Spark to deliver data pipelines • Experience of using Terraform to deliver infrastructure as code • Experience of developing ‘in the open’ - using tools such as Github • Experience of content design to enable users to perform technical tasks Nice to Haves • Demonstrate an understanding of Hackney’s approach to technology and data as set out on https://playbook.hackney.gov.uk • Demonstrate how they provide opportunities for apprentices • Experience of business intelligence tools - particularly Qlik • Knowledge or experience of machine learning Work Location Remotely, and in Hackney Council offices in Mare Street, E8 where face to face contact is required Working Arrangments This work can be done remotely but will require fortnightly reporting (e.g. via show & tells) to involve key stakeholders and regular engagement with practitioners through video conferences. The current team have also found some face-to-face working beneficial. We will provide access to key systems to project team members where required. No. of Suppliers to Evaluate 5 Proposal Criteria • Understanding of the brief • Clarity of the approach • Experience from a similar project • Team structure, including skills, experiences and relevance of individuals • Identification of risks and plan to mitigate them • Value for money Cultural Fit Criteria • Work as a team with our organisation and other suppliers • Be transparent and collaborative when making decisions • Share knowledge and experience with team members and the wider service • Hackney Council is a diverse organisation, reflecting the diversity of our borough. Describe how you would successfully work as a team with Hackney and other suppliers in a diverse environment. Payment Approach Time and materials Assessment Method Presentation Evaluation Weighting Technical competence 60% Cultural fit 5% Price 35% Questions from Suppliers 1. If successfully shortlisted, what will be the deadline for submitting a written proposal? We expect the deadline for written proposals to be 9am on Monday, 20 December. 2. Are MadeTech the current incumbent and likely to be bidding on this also? As stated in the brief, the first phase of developing our data platform was done in partnership with MadeTech. This is an open tender and open to any bidder. Any incoming supplier will have access to documentation about the current platform as well as all tools and systems used in the project. The IP is owned by Hackney and, in addition, Hackney officers have been very engaged in the project and will be able to provide handover. 3. How often (days per week) would you like the chosen supplier team to attend (or work form) the Hackney office? Face-to-face working is not a requirement, but something that the current team have found useful. The maximum we would expect would be one day per week, but arrangements with the successful supplier will be flexible and take into account the risk of Covid-19. 4. ’Correct tools and processes for ingestion, transformation, and analysis”.To what extent have tools and processes been already decided?Is the context of this ask to review and change or to extend?What is the scope regarding tools and processes? We have a number of tools and processes in place within AWS (ingestion, transformation) and outside of it (analysis in BI tools, notebooks; data catalogue). However, in some cases these are incomplete or untested with our users. We want to review these as we bring more users onto the platform and develop our understanding of the user journey and how tools and processes work together. We expect that this would lead to iteration and extension of our current offer, but we are open to change if this is necessary to meet user needs.