About this role
The opportunity
We are recruiting a Cloud Data Platform Engineer to take ownership of the technology environment supporting our data engineering, analytics and machine learning teams.
Your focus will be on making the platform dependable, secure and efficient. You will combine hands-on engineering with operational ownership, helping technical teams deliver their work without unnecessary infrastructure or access barriers.
Working alongside engineering and analytics colleagues, you will develop the platform’s capabilities, strengthen its controls and improve the way it is deployed, monitored and maintained.
What you will be responsible forPlatform engineering and automation
Manage Databricks environments and the compute resources used for data processing, SQL analysis and modelling.Develop repeatable infrastructure deployments using Terraform and version-controlled configuration.Establish automated release processes, ensuring platform changes are tested, reviewed and deployed consistently.Maintain reusable configurations and engineering standards that simplify the setup of new workloads.Support workflow scheduling and integrations with the wider data ecosystem. Security and data governance Administer Unity Catalog and maintain appropriate permissions across data assets and environments.Configure identities, service accounts, authentication integrations and secrets management.Work with relevant colleagues to implement network controls, encryption and secure access.Maintain the controls and records needed to support auditing, data traceability and compliance requirements. Performance and operational management Investigate and improve the performance of Spark workloads, Delta tables and SQL compute.Review resource consumption and identify ways to reduce unnecessary platform and cloud expenditure.Set up monitoring and alerting to identify failures, performance issues and capacity constraints.Resolve platform incidents, document their causes and introduce improvements that prevent recurrence.Maintain operational documentation and review changes for quality and reliability. Supporting analytics and machine learning Provide the platform foundations needed for data science experimentation and production modelling.Support capabilities such as MLflow, feature management and model deployment.Help users diagnose platform-related problems and adopt consistent engineering practices.Contribute to technical reviews, knowledge sharing and mentoring across the team. What we are looking for Practical experience managing Databricks in a production environment.A strong understanding of Apache Spark, Delta Lake and Unity Catalog.Experience working with one or more major cloud providers, including storage, networking and identity management.Confidence using Terraform and Git-based deployment pipelines.Good Python and SQL skills, with experience using PySpark and scripting for automation.Experience diagnosing performance issues and optimising distributed data workloads.Familiarity with single sign-on, automated identity provisioning and secure credential management.An understanding of data protection requirements and security controls within a commercial environment.
Working arrangements
This is a permanent, full-time position with hybrid working and two days per week in a London office. Occasional UK travel may be required. Applicants must have the right to work in the UK.