Now hiring

Databricks Data Engineer @ GIS Recruiter Ltd

London, England, GBOnsiteFull-time
Apply with ResuMinder

Opens on the employer's site

About this role

Salary: £75,000 - 95,000 per year

Requirements

  • 5+ years of hands-on Databricks experience, including expert-level knowledge of Azure Databricks.
  • Advanced PySpark, Spark SQL, and Python programming skills.
  • Strong experience developing complex ETL/ELT pipelines and implementing Medallion Architecture (Bronze, Silver, and Gold).
  • Deep understanding of Delta Lake, Delta Tables, and Delta Live Tables / Lakeflow Declarative Pipelines.
  • Strong knowledge of Apache Spark architecture, optimisation, and performance tuning.
  • Hands-on experience with Unity Catalog, access controls, and data governance.
  • Experience implementing incremental data loads, CDC, and SCD Type 1/Type 2.
  • Strong SQL development and query optimisation skills.
  • Experience integrating Azure Data Factory with Databricks and knowledge of Azure Data Lake Storage Gen2.
  • Experience with CI/CD pipelines, Git, Azure DevOps, and automated deployments.
  • Experience implementing data quality, validation, and reconciliation frameworks.
  • Previous experience delivering data engineering projects in UK Government or other public sector organisations, such as Central Government, Local Government, NHS, DWP, HMRC, or the Home Office.
  • Understanding of public sector data governance, security, and information management.
  • Experience working with sensitive and confidential datasets.
  • Knowledge of UK GDPR, the Data Protection Act 2018, and relevant government data security standards.
  • Experience working in Agile delivery teams and government digital transformation programmes.
  • Eligibility and willingness to undergo SC clearance, government security vetting, and background checks.
  • Previous UK Government or Central Government project experience is highly desirable.

Responsibilities

  • Design and develop scalable Databricks data pipelines.
  • Build and maintain enterprise-grade Lakehouse platforms.
  • Optimise Spark workloads and Databricks clusters.
  • Implement robust data quality and governance controls.
  • Collaborate with architects, analysts, and business stakeholders.
  • Support production deployments and troubleshoot complex data issues.
  • Ensure solutions meet public sector security and compliance requirements.

Technologies

  • Azure
  • CI/CD
  • Cloud
  • Databricks
  • DevOps
  • ETL
  • Git
  • Support
  • Python
  • PySpark
  • SQL
  • Security
  • Spark
  • Unity
  • GameDev

More

We are seeking an experienced Senior Databricks Data Engineer to deliver enterprise-scale data engineering solutions using Azure Databricks. The role involves building complex data platforms within UK public sector environments, working with large datasets, sensitive information, and secure cloud-based data architectures.

last updated 47 week of 2026

Skills

DatabricksAzure DatabricksPySparkSpark SQLPythonETLELTMedallion ArchitectureDelta LakeDelta TablesDelta Live TablesUnity Catalog

Ready to apply?

Install the ResuMinder extension and we'll auto-fill the application in seconds — no rewriting.

See how your CV scores