Analytically ready MCA data

Description

Summary of the work Creation of analytically ready data for MCA Data Analysts, Data Scientists to drive more efficient and effective operations, measure policy and inform decision making, taken from our operational systems into the MCA Cloud Data Platform. Expected Contract Length 6 months Latest start date Monday 1 November 2021 Budget Range We are looking to make analytically ready data available from up to 15 sources and are expecting an average of £10,000 per data source. Our total budget is £125k ex VAT. Why the Work is Being Done The organisation is undergoing a data transformation with increasing adoption of self-serve analytics (using Power BI as the toolset), and increasing use of data for advanced analysis to support operational and strategic decision making. To support this the organisation has invested in the technology to create the MCA Cloud Data Platform (using Azure services) to access and share data from our core operational systems, and are looking for support to ingest and transform the data to make it ready for analytical use. Problem to Be Solved Access to analytically ready data for MCA Data Analysts, Data Scientists to drive more efficient and effective operations, measure policy and inform decision making. Who Are the Users As an MCA Power BI Pro User I want access to trusted data so that I can create analytical reports to ensure that my operational activities are efficient, effective and have the best outcomes. As an MCA Data Scientist I want access to data from multiple MCA systems so that I can combine it to derive new insights and drive our policies and strategic decisions. Early Market Engagement None Work Already Done The organisation has already built the technology to support ingestion and analysis of data in the MCA Cloud Data Platform. We have also been working with a number of our operational system support partners to create APIs for access to the data, this work is ongoing. Existing Team The supplier will be working with the Data Engineering, Data Science and Analytics and Data Architecture teams, as well as our operational system support partners. Current Phase Not applicable Skills & Experience • Have experience in Data Pipeline Design and Development • Have experience in Data Platform Design • Have experience with Azure Cloud Services • Have experience in creating Analytically Ready Data • Have experience with implementing good data management practises. Nice to Haves • Have experience in analysing Maritime Data • Have experience working with AIS (Automatic Identification System) Data • Have experience in implementation of Data Streaming Pipelines • Have experience in training technical users. Work Location Spring Place, 105 Commercial Road, Southampton, SO15 1EG Working Arrangments Virtual via MS Teams, bi-weekly workshops in our HQ, Spring Place, Southampton. Security Clearance Basic Security Checks Additional T&Cs You should use the standard contract unless your organisation needs to include additional terms and conditions. Read about how the terms and conditions work: https://www.gov.uk/guidance/terms-and-conditions-of-digital-marketplace-frameworks No. of Suppliers to Evaluate 6 Proposal Criteria • technical solution • approach and methodology • how the approach or solution meets user needs • how the approach or solution meets your organisation’s policy or goal • estimated timeframes for the work • how they’ve identified risks and dependencies and offered approaches to manage them • team structure • value for money Cultural Fit Criteria • work as a team with our organisation and other suppliers • be transparent and collaborative when making decisions • have a no-blame culture and encourage people to learn from their mistakes • take responsibility for their work • share knowledge and experience with other team members • challenge the status quo • be comfortable standing up for their discipline • can work with clients with low technical expertise Payment Approach Fixed price Assessment Method • Case study • Work history • Reference • Presentation Evaluation Weighting Technical competence 70% Cultural fit 10% Price 20% Questions from Suppliers 1. Regarding Nice to have skills, will SME with no maritime skills be able to bid fairly for the project ? We will be looking for suppliers who can meet all of the essential skills and experience. If suppliers are also able to show evidence of meeting the nice to have skills then this will be included in their overall score. This should not preclude any supplier without maritime experience that meets the essential skills and experience criteria from bidding fairly. 2. Can you tell us how you arrived at the budget for the 15 data source feeds that you require? The budget provided is based on previous costs for similar work and is shown as an indicative average. It is recognised that some data sources will be more complex than others and that suppliers may need to make assumptions about the data sources. 3. The bi-weekly meetings in Southampton is ambiguous, do you mean once every two weeks or twice a week? We would like to have the option for workshops with the supplier in our headquarters in Spring Place, Southampton once every two weeks. We will still consider suppliers offering exclusively remote working. 4. Given that the authority anticipates a cost of £10,000 per data source, is the requirement more about the provision of those up to 15 data sources or setting the solid foundations for that provision? We currently have high level designs for ingestion patterns of batch processing for these sources of data. We are looking for the supplier to utilise, and if necessary extend, these designs to implement the data pipelines to provide analytically ready data for our users.We are also looking for the supplier to extend the current implementation to include stream processing for two of the data sources, AIS Vessel Tracking and Incidents from our Vision Command and Control system to provide near-real time reporting. 5. Would you be willing with a remote (UK-based) team? We would like to have the option for workshops with the supplier in our headquarters in Spring Place, Southampton once every two weeks. We will still consider suppliers offering exclusively remote working. 6. Please could you provide more detail on which technology choices you have made on the Azure platform as part of the MCA Cloud Data Platform. For example should we assume use of Data Factory for ETL? The MCA Cloud Data Platform utilises Azure services to bring MCA Data into a central repository, integrated and transformed with tooling for advanced analysis and data science to derive new actionable insights.Data is stored within Azure Data Lake and moved through three tiers (Raw, Clean, Conformed). Data Movement is carried out using Azure Data Factory and Azure Logic Apps, with data transformations being carried out using Databricks.The data is then made available in the Consumption layer through connection to SQL Server, via a Data Science Virtual Machine or through creation of APIs. 7. Are you able to identify the types of data which will be analysed?And provide an example Use Case? The data sources cover the following themes:• Incidents• Vessels• Wrecks• Seafarer Certification• MCA InternalMCA has been rolling out self-serve analytics through use of PowerBI and providing trusted datasets to users to build their reporting. The majority of these data sources are internal system where teams would like to monitor their KPIs, understand data quality, provide periodic reporting to stakeholders and answer adhoc questions from external stakeholders. 8. Can you provide further information about each of the data sources? There are 15 data sources identified as required for analysis, covering the following themes: Incidents Vessels Wrecks Seafarer Certification MCA InternalThere is a blend of source systems for the data which are accessible through the following: Restful JSON APIs. On Premise SQL Server Database Cloud Hosted SQL Server Database Sharepoint CSV Files.We have high level designs for ingestion patterns of batch processing for these sources of data. We are looking for the supplier to extend these to include stream processing for two of the data sources, AIS Vessel Tracking and Incidents to provide near-real time reporting. 9. Is the work purely for a team to build the pipelines and create the data transformation code on your platform or, are you looking for (and will accept) a technical solution to work with your MCA platform (to accelerate the pipeline creation), as well as resources to build the pipelines? We would evaluate the proposal of any supplier that meets the essential skills and experience, and will accept solutions that extend and/or complement our current MCA Data Platform architecture. 10. Is MCA willing to work with a vendor which provides clean/de-dup’ed data per source using its own tooling or is this requirement for a contractor type services to build code in an MCA environment? We would evaluate the proposal of any supplier that meets the essential skills and experience, and will accept solutions that extend and/or complement our current MCA Data Platform architecture, including use of new tools. 11. Regarding “Nice-to-have skills- Have experience in analysing Maritime Data”. Can you please explain if ‘experience in Maritime Data’ will be used to exclude SME with no Maritime background but has all the relevant Digital skills to deliver an outcome ? Answer to above will save valuable time. We will be looking for suppliers who can meet all of the essential skills and experience. If suppliers are also able to show evidence of meeting the nice to have skills then this will be included in their overall score. This should not preclude any supplier without maritime experience that meets the essential skills and experience criteria from bidding fairly. 12. Please could the MCA confirm whether it requires the supplier to develop dashboards as well as implement the data pipelines? The supplier will be required to provide analytically ready data that our Power BI Pro Users and Data Scientists can use to meet their needs. The development of the dashboards and analysis will be carried out by MCA. 13. What is the scope of onsite meetings – is it to cover project management/progress only, or will these meetings be working sessions/workshops? We would like to have the option for workshops with the supplier in our headquarters in Spring Place, Southampton once every two weeks. This would be for technical demos, knowledge transfer sessions, design workshops etc. We will still consider suppliers offering exclusively remote working.