About this role
Salary: £59,000 - 99,000 per year
Requirements: Bachelors or Masters degree, or equivalent experience, in Computer Science, Engineering, Information Systems, or another numerate field.10 years of experience as a Data Engineer with a proven track record of leading and mentoring teams.Experience building market data solutions within a trading or financial services environment.In-depth expertise in Python, including Pandas, web scraping libraries such as Selenium, Beautiful Soup, and Requests, and machine learning libraries such as TensorFlow and PyTorch.Generative AI techniques including agents, skills, prompt engineering, RAG, AI SDLC, context engineering, and evaluations.Strong SQL expertise, including advanced optimisation techniques, and experience with relational databases, preferably PostgreSQL, including data modelling and design.Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data.Proven experience with Apache Airflow for data orchestration and scheduling at scale.Expertise in AWS cloud services such as S3, Redshift, EC2, and Lambda.Experience defining and designing the full SDLC, including development, test, and deployment, using Git source control and ADO or Jenkins pipelines.Experience working in a Linux environment and proficiency with Git version control.Excellent written and verbal communication skills, with the ability to present complex technical concepts clearly to technical and non-technical audiences.Demonstrated experience working in a global environment and building strong relationships across borders.A passion for building high-performing teams and fostering a culture of innovation and continuous learning.Desirable: experience with Databricks for large-scale data processing.Desirable: experience with advanced data visualisation tools such as Tableau or Power BI with a focus on insightful storytelling.Desirable: knowledge of Microsoft SQL Server administration and query performance tuning.Desirable: familiarity with core cloud concepts such as subscriptions, resource groups, and IAM.Desirable: knowledge of the commodities front office domain, including financial and physical instruments and market data prices. Responsibilities: Establish and lead a high-performing, global data engineering squad, fostering a collaborative, results-oriented environment and owning the technical direction for the teams deliverables.Design, develop, and implement highly scalable and efficient data pipelines using advanced Python, SQL, and Apache Airflow.Lead the extraction, transformation, and loading of large and complex datasets, including time series data, from sources such as web APIs, databases, and real-time streams.Leverage Generative AI platforms and frameworks to deliver novel capabilities for the trading organisation.Use AWS cloud services to build, deploy, and manage data pipelines in a secure and cost-effective manner, with a focus on cloud cost optimisation and Total Cost of Ownership.Coach and mentor a team of 8 to 10 data engineers, providing guidance on standard processes, technical development, and career growth.Lead and collaborate within a global, cross-functional team, building strong relationships across borders and time zones.Drive the adoption of Agile methodologies and use Azure DevOps to plan, track, and manage team workload efficiently.Create compelling data visualisations using advanced tools to communicate insights to both technical and non-technical audiences.Champion automation tools and standard processes to deliver reliable, repeatable results, ensuring configuration is maintained under source control with automated CI/CD deployment and testing.Develop and maintain clear, concise technical documentation for data pipelines and procedures.Continuously improve through agile sprint retrospectives, RCAs, and design review workshops, coaching junior team members and using data to guide team investment decisions. Technologies: AIAirflowAWSLambdaRedshiftAzureCI/CDCloudDatabricksDevOpsEC2GitIAMJenkinsLinuxMachine LearningPostgreSQLPower BIPyTorchPythonRAGSQLSeleniumTableauTensorFlowWebpandasETLSupportKanbanNoSQLSecurity More:
We are BP, supporting our people to grow in a diverse and exciting environment. We believe our team is strengthened by diversity and offer benefits that help work fit with life, including flexible working options, a generous paid parental leave policy, and excellent retirement benefits. This role supports our Trading Analytics and Insights organisation within our Trading and Shipping business, with a hybrid office/remote working arrangement in London, United Kingdom. No travel is expected, and the role is not eligible for relocation. We also provide reasonable accommodations for individuals with disabilities during the application or interview process and for crucial job functions. We are an equal opportunity employer and do not discriminate on the basis of protected characteristics.
last updated 36 week of 2026