Stotles logo
Closed

Build Enterprise Data Warehouse in the Cloud Project

Published

Description

Summary of the work Develop a cloud-based Enterprise Data Warehouse solution using the Microsoft Azure Data Platform where the key sources are Workday and Microsoft Dynamics CRM. Expected Contract Length implementation and early life support expected to last for no longer than 12 – 24 months Latest start date Saturday 15 May 2021 Why the Work is Being Done The Ombudsman Service has an existing Enterprise Data Warehouse (“EDW”) that was built in 2014 to provide the data to inform most of our internal and external reports. The data for within the existing EDW is stored in a Microsoft SQL Server database that is hosted in our own data centre. SAP Data Services is used to extract raw data from various source systems, before transforming it and loading it into the EDW. The Ombudsman Service is in process of exiting its own data centres and migrating services to the cloud. To this end, the Ombudsman Service wishes to build a new cloud-based EDW. This project covers the design, build, test, go-live and early life support of a new cloud based EDW built and stored on the Microsoft Azure Data Platform. Problem to Be Solved You will design, build and test a new EDW for us on Microsoft Azure Data Platform. This will include interfacing to approximately fifteen of our systems to extract data to load into the new EDW. We will be looking for the data to be refreshed on a nightly basis with checkpoints built in to ensure reliability of data loads and easy restart after an issue. We are in the process of implementing Workday for Finance and HR and expect this to go-live by September 2021. The EDW must be in place for the Workday feed by then as Workday is an important source of data. Following this, our core business system is built on Microsoft Dynamics 365 CRM, and this would be implemented following the Workday solution along with the remaining systems. Who Are the Users Users will not use the EDW directly. Power BI will be used to create reports and dashboards for our end users to consume. You will be responsible for integrating with Power BI, but not for creating reports or dashboards. We have an internal BI Team who support Power BI reports and dashboards now and going forward. Early Market Engagement None. Work Already Done We have worked with Microsoft on a PoC to understand the technology required and see how it would work for us. The technology used has been the Microsoft Azure Data Platform, including Azure Synapse Analytics, Azure Data Factory and Azure Data Lake. Existing Team There is an existing ETL team in-house who support the current on-premise solution which has been developed using SAP Data Services. Current Phase Discovery Skills & Experience • Must have relevant experience of delivering a data warehouse on Microsoft Azure Data Platform. (Describe a recent Azure Data Platform data warehouse project. What, where, when, duration, result.) • Must have experience of overcoming challenges delivering a data warehouse on Microsoft Azure Data Platform. (Describe your most significant Azure Data Platform warehouse delivery challenge and how you overcame it.) • Must have Azure Synapse Analytics and Azure Data Factory skills. (Describe your skills with Azure Synapse Analytics and Azure Data Factory.) • Must have experience of delivering a data warehouse where the daily data load is time sensitive. (Describe a data warehouse delivery where the daily load completes by a set time.) • Must have experience of deploying a suitably structured team to deliver a data warehouse using Microsoft Azure Data Platform. (Set out indicative roles and team structure.) • Must have experience of delivering a data warehouse where Microsoft Dynamics CRM was a significant source of data. (Set out what, where, when, result.) • Must have a suitable methodology for designing an Enterprise Data Warehouse. (Please can you describe the methodology you would use in the design of the Enterprise Data Warehouse?) • Must have experience in designing an enterprise data warehouse that integrates well with a reporting tool such as Power BI. (Please can you describe how you have achieved this.) • Must have experience of rapid delivery. (Describe when you have delivered a data warehouse solution within five months to be ready for another project delivering at that time?) Nice to Haves • It would be good if the supplier has Azure Data Engineer Associate certifications. (How many people are certified and how many of those are permanent employees of your company?) • It would be good if the supplier has Azure Database Administrator Associate certifications.  (How many people are certified and how many of those are permanent employees of your company?) • It would be good if the supplier has evidence of guaranteeing the design and build of a data warehouse, where the ongoing support of such is provided by another party. • It would be good if the supplier has experience of designing data warehouses where the data can be updated on multiple occasions throughout the day. (State what, where, when, result.) • It would be good if the supplier has experience of delivering a data warehouse where Workday is a significant data source. (Describe a delivery where Workday is a significant source.) • It would be good if the supplier would underwrite the processing cost and performance of the enterprise data warehouse. (Describe experience of underwriting warehouse performance and cloud computing costs.) • It would be good if the supplier has delivered data warehouses where Microsoft Data Lake hasn’t been used. (Describe delivering a warehouse on Azure but not using Microsoft Data Lake?) Work Location Initially, work will take place remotely due to the Covid epidemic. As soon as practicable, work will take place at Exchange Tower, London, E14 9SR. Working Arrangments Initially, work will take place remotely due to the Covid epidemic. As soon as practicable, work will take place at Exchange Tower, London, E14 9SR. Additional T&Cs A completed Call Off including Order Form will be shared with the shortlisted suppliers and form part of the additional assessment. No. of Suppliers to Evaluate 6 Proposal Criteria • How well the proposed solution meets requirements as set out in the work order. (35%) • Confidence that the delivery will be completed within the required timeline. (15%) • The proposed delivery approach and methodology (15%) • How the supplier identified risks and dependencies and offered approaches to manage them (15%) • Team structure (10%) • Case Study (5%) • Reference (5%) Cultural Fit Criteria • Take responsibility for the delivering a successful outcome for the Ombudsman Service • Adopt a “one-team” culture. • Be transparent and collaborative when making decisions • Have a no-blame culture and encourage people to learn from their mistakes • A culture of quality assurance is embedded in the team • Be challenging of the Ombudsman Service where appropriate • Share knowledge and experience with other team members Payment Approach Fixed price Assessment Method • Case study • Reference • Presentation Evaluation Weighting Technical competence 60% Cultural fit 15% Price 25% Questions from Suppliers 1. Can Financial Ombudsman Service please advise the budget for this project? We are not in a position to provide budget information at this time. 2. Can Financial Ombudsman Service please advise the budget for this project? We are not in a position to provide budget information at this time. 3. Are there any third-party suppliers involved withing the existing ETL or SAP services? Our existing on-prem data warehouse is supported by our internal team. 4. Are the Authority open to onshore-offshore model? with key resources at Onshore and development work at offshore? We are open to considering the most efficient options for delivery of our requirements. This includes offshore delivery where our data protection and information security needs can be met. 5. Can you please advise the timelines of this procurement (e.g. shortlisted companies notified, anticipated contract award)? Also, can you confirm if shortlisted suppliers will have the opportunity to present? We expect to notify shortlisted companies by the middle of February 2021 with a proposal phase to start shortly thereafter. We would expect to announce award a contract in April 2021 with delivery to commence no later than 15th May 2021 (ideally sooner). 6. Who from your team will be scoring applications for this opportunity and what positions do they hold? Our evaluation team is comprised of representatives from our IT and procurement organisations. 7. Would you be happy for the candidate to be based outside the UK? We are not seeking a candidate, we are seeking a team to deliver an outcome. We are open to considering the most efficient options for delivery of our requirements. This includes offshore delivery where our data protection and information security needs can be met. 8. is there a current incumbent? Our existing on-prem data warehouse is supported by our internal team. 9. What tool do you currently use for ETL and what tool do you plan to use on Azure for ETL? We currently use SAP Data Services, but will not be going forward using that tool once the new EDW in the Cloud is live. We are looking for Data Factory, which is part of the Microsoft Azure Data Platform, to be the ETL tool used in the new solution. 10. Do you have an active Azure subscription? Yes, we do. 11. Please could you clarify whether you are asking if we have we delivered a data warehouse solution without using Microsoft Components in Azure e.g. delivering a Snowflake data warehouse solution hosted on a Azure platform or similar to this? We are asking whether you have experience of delivering a data warehouse on Microsoft Azure Data Platform. 12. Please could you please clarify whether you are referring to the data source being on-premise or online? Data sources will be both cloud systems and on-prem systems. 13. Can you please clarify what you mean by underwriting the cost and performance of the data warehouse? will you be expecting the supplier, for instance, to take on consumption costs of the data warehouse if it breaches a certain threshold, for example? We mean that you should state experience of where you have underwritten cloud computing run costs associated with a data warehouse implementation. We recognise that efficient configuration of ETL procedures can lead to lower ongoing cloud computing costs and in the proposal phase we are interested in exploring the suppliers' appetite to own some or all of this commercial risk. 14. Would you consider using a provider’s Azure platform to kick start the project while all the components are being set up in your environment? We are interested in the most efficient delivery solution. We understand that configuring the Azure platform is relatively straightforward, but are keen to understand any benefit of this model (we will do this in the proposal phase). Our information security and data protection requirements would of course need to be met. 15. What type of skills do you already have in-house and expect to have a role in this project? e.g. data architects, deployment engineers This engagement is as a fixed price for a fixed outcome and we therefore expect the supplier to provide all roles necessary to deliver the outcome. We will of course provide subject matter experts to provide information as it relates to our Azure deployment, other relevant infrastructure and the data sources. 16. Did the POC produce relevant documentation e.g. data dictionary, source to target mappings, logical data model, architecture diagrams etc.? The PoC produced an architecture diagram of the potential solution, but did not produce other detail that has been mentioned. The purpose of the PoC was to understand the technology that would be used and establish that it could provide what we require. 17. Did the team define an Operating and Support Model as part of the POC? Early life support is included in this opportunity. We are expecting the supplier to propose an efficient model for this during the proposal phase. 18. Did the POC include any Data Management & Governance elements? The PoC focused on the Microsoft Azure Data Platform. 19. Which data sources as part of the 15 data sources were ingested in the POC? The PoC focused on loading data from our Casework system that is built on Dynamics 365. 20. If a data model was delivered, did it include all 15 data sources? No, a data model was not delivered as part of the PoC. 21. Did the POC deliver a repeatable ingestion pattern? The PoC developed a flow, but it could not be taken forward to be part of this implementation. 22. Did the POC deliver any re-usable components that can be leveraged for a production build? No 23. What is the recommended deployment approach? Are there any polices in place around the DevOps, infrastructure automation and test automation that could be leveraged for cloud applications? We are looking for the suppler to recommend a deployment approach during the proposal phase. 24. What SLAs are for batch process in current EDW? How much time does longest running job takes? Are there any performance bottlenecks currently? We don't think that performance of the existing EDW will have any bearing on the new cloud based EDW. We are seeking the supplier to design the new EDW to meet our requirements in the proposal phase. 25. Will consumption be only via PowerBI or are there any other tool or application? Power BI will be the main consumer of the data in the EDW. However, there is also an update to Microsoft MDS (Master Data Services), as well as various Excel and CSV outputs. 26. Are there any data management and data governance policies in place that the project team need to adhere to? Yes, we will share more detailed requirements with the shortlisted suppliers in the proposal phase. 27. Is migration of current EDW in scope (both for data & processes)? Will current EDW be decommissioned in future? Yes we expect the new EDW deployment to be more of a cutover than a migration. We plan to decommission the existing EDW after a suitable period. 28. Are Azure foundation services established, which are pre-requisite for Azure EDW build? We already use Microsoft's Azure. We will need to turn on the Microsoft Azure Data Platform before use. 29. In Q16 regarding ‘Microsoft Data Lake’, which Azure services are covered under ‘Microsoft Data Lake’? With Microsoft Data Lake, we are meaning just the storage part. We are looking to use Microsoft Data Lake rather than Azure Data Blob. 30. What does the buyer mean by “Underwrite the processing cost and performance of the enterprise data warehouse.” We mean that you should state experience of where you have underwritten cloud computing run costs associated with a data warehouse implementation. We recognise that efficient configuration of ETL procedures can lead to lower ongoing cloud computing costs and in the proposal phase we are interested in exploring the suppliers' appetite to own some or all of this commercial risk.

Timeline

Publish date

3 years ago

Close date

3 years ago

Buyer information

Explore contracts and tenders relating to Financial Ombudsman Service

Go to buyer profile
To save this opportunity, sign up to Stotles for free.
Save in app
  • Looking glass on top of a file iconTender tracking

    Access a feed of government opportunities tailored to you, in one view. Receive email alerts and integrate with your CRM to stay up-to-date.

  • ID card iconProactive prospecting

    Get ahead of competitors by reaching out to key decision-makers within buying organisations directly.

  • Open folder icon360° account briefings

    Create in-depth briefings on buyer organisations based on their historical & upcoming procurement activity.

  • Teamwork iconCollaboration tools

    Streamline sales workflows with team collaboration and communication features, and integrate with your favourite sales tools.

Stop chasing tenders, start getting ahead.

Create your free feed

Explore other contracts published by Financial Ombudsman Service

Explore more open tenders, recent contract awards and upcoming contract expiries published by Financial Ombudsman Service.

Explore more suppliers to Financial Ombudsman Service

Sign up