About this role
Senior Data Engineer – Python, Databricks & Counterparty Credit Risk
Location: London – Hybrid
We are recruiting for an experienced Senior Data Engineer to join a leading global banking organisation, working across a highly technical data environment supporting Counterparty Credit Risk (CCR), Credit Risk and wider risk data.
This is a hands-on engineering position requiring strong expertise across Python, PySpark, SQL and Databricks, alongside experience designing, building and optimising scalable data pipelines and Lakehouse solutions.
Candidates should ideally have previously worked with Counterparty Credit Risk or Credit Risk data, including areas such as counterparty/exposure data, risk analytics, regulatory risk or associated banking risk datasets.
Key Responsibilities
Design, develop and maintain scalable data pipelines using Python, PySpark and DatabricksBuild and optimise ETL/ELT workflows using Spark and Delta LakeEngineer and transform data supporting Counterparty Credit Risk, Credit Risk and wider risk functionsWork with datasets relating to areas such as counterparties, credit exposures, limits, collateral and risk analyticsDesign robust data models and architectures across Lakehouse / Medallion environmentsDevelop and manage Databricks notebooks, jobs and workflowsOptimise large-scale distributed data processing and troubleshoot Spark performanceImplement strong data quality, reconciliation, governance and monitoring controlsBuild reliable logging and alerting across production data pipelinesIntegrate data platforms with Power BI dashboards and wider applicationsSupport CI/CD and engineering best practices across development, testing and production environmentsWork closely with Risk, Technology, Analytics and other senior stakeholders Required Experience
10+ years' experience across Data Engineering or closely related data disciplinesStrong hands-on Python development experienceStrong PySpark / Apache Spark engineering experienceExtensive hands-on experience with Databricks, including notebooks, jobs/workflows and Delta LakeAdvanced SQL and relational database experienceStrong track record designing and building scalable ETL/ELT data pipelinesExperience with Delta Lake, Lakehouse and/or Medallion architecturesExperience handling banking risk data, ideally within Counterparty Credit Risk (CCR), Credit Risk, counterparty exposure or closely related risk domainsStrong understanding of data modelling, data quality and production data engineeringCloud experience across Azure, AWS and/or GCPExperience with modern data platforms/warehouses such as Snowflake, BigQuery or RedshiftExperience integrating data platforms with Power BIExperience with orchestration such as Databricks Workflows or Airflow Highly Desirable
Direct experience engineering data for Counterparty Credit Risk, including exposure/counterparty datasetsKnowledge of areas such as PFE, EPE, EAD, CVA/XVA, SA-CCR, collateral, netting or counterparty limitsPrevious experience within an investment bank, global bank or capital-markets environmentDatabricks Data Engineer certificationKafka / Structured Streaming experienceDelta Lake CDC, Time Travel and optimisationCI/CD, Git and DevOps experienceDocker / Kubernetes experienceExperience exposing data through APIs or supporting React-based front-end applicationsExperience supporting analytics or ML workloads within Databricks We are particularly interested in genuinely hands-on Senior Data Engineers who can demonstrate the Databricks and Python solutions they have personally designed, built, optimised and supported in production.
If you have strong Databricks, Python and PySpark engineering experience alongside exposure to Counterparty Credit Risk, Credit Risk or banking risk data, please apply for further information.