About this role
Salary: £? - ? per year
Requirements
- Exceptional hands-on Python, PySpark and Apache Spark expertise, including independently writing, debugging and optimising complex production code
- Deep understanding of Spark internals and performance optimisation, including execution plans, joins, shuffles, partitioning, memory management and data skew
- Strong AWS data engineering experience, particularly with AWS Glue
- Strong Apache Airflow experience
- Excellent software engineering fundamentals, including code quality, testing, performance and maintainability
- Experience designing and delivering large-scale data pipeline migrations or modernisation programmes
- Experience operating at Staff, Principal or Senior Lead Engineer level, or demonstrably performing at that standard
- Ability to assess complex engineering challenges independently, challenge existing approaches and shape scalable solutions
- Strong architectural and conceptual understanding, supported by hands-on delivery experience
- Demonstrable experience using AI-assisted development tools, such as Claude or GitHub Copilot, to accelerate software engineering delivery
- Ability to critically review, validate and improve AI-generated code
- Experience influencing engineering practices and technical direction across multiple teams
- Beneficial experience with Snowflake, legacy ETL modernisation and migration, reusable migration frameworks or automation, and financial services or other complex, highly regulated environments
Responsibilities
- Provide Staff-level technical leadership across complex data engineering and modernisation initiatives
- Build, prove, debug and optimise engineering solutions hands-on using Python, PySpark and Apache Spark
- Design and optimise scalable data pipelines and processing frameworks across AWS
- Address complex Spark performance challenges, including shuffles, joins, data skew, partitioning and memory optimisation
- Design and challenge engineering approaches across AWS Glue, Apache Airflow and associated technologies
- Review existing code, identify performance bottlenecks and drive improvements in engineering quality
- Establish reusable engineering patterns, frameworks and approaches to accelerate large-scale data migrations
- Apply AI-assisted development tools to improve engineering productivity, code quality and delivery speed
- Challenge technical assumptions, influence architectural decisions and raise engineering standards across multiple teams
Technologies
- AI
- Airflow
- AWS
- AWS Glue
- Copilot
- ETL
- GitHub
- Python
- PySpark
- Snowflake
- Spark
- Cloud
More
Were supporting a major UK financial services organisation through a significant data engineering modernisation and transformation programme. The work includes modernising a large-scale enterprise data estate, migrating and transforming thousands of data pipelines, developing new data products, and adopting AI-assisted engineering practices. This is a hands-on Staff-level contract role for someone who can set technical direction while continuing to build and optimise production-quality code. The initial contract is six months, with a rate of up to £1,100 per day inside IR35. The role is hybrid in London, with three days per week on-site.
last updated 46 week of 2026