About this role
Salary: £50,000 - 50,000 per year
Requirements: Experience developing and managing PySpark workflows within AWS environments.Knowledge of AWS Lambda and event-driven solutions to support automated data processing.Experience provisioning and supporting cloud infrastructure using Amazon EC2 and automation tools.Understanding of Agile ways of working and experience contributing to CI/CD delivery practices.Ability to optimise data processing, storage, and queries to improve performance and data quality.Experience designing and automating ETL pipelines using PySpark and Spark SQL.Strong full stack development background with technologies including .NET 4.8, C#, JavaScript, C++, and Java. Responsibilities: Help deliver modern data engineering solutions that support analytical reporting, business intelligence, and strategic decision-making.Design, build, maintain, and optimise scalable data products used by analysts and data scientists.Develop and orchestrate PySpark-based ETL pipelines using Amazon EMR, AWS Lambda, and EC2.Process large volumes of structured and unstructured data efficiently in a hybrid cloud environment.Collaborate with stakeholders to translate reporting requirements into robust technical solutions.Maintain high standards of data quality, performance, and security.Contribute to Agile delivery practices and support CI/CD automation.Maintain technical documentation, conduct code reviews, and share knowledge with other engineers and data professionals. Technologies: AWSLambdaBusiness IntelligenceC#CI/CDCloudEC2ETLSupportJavaJavaScriptPySparkSQLSecuritySparkASP.NET More:
We are DWP Digital, part of the UKs largest government department, and we use fresh ideas and leading-edge technology to build and maintain digital solutions used by nearly every person in the UK at key moments in their lives. Our Management Information team works in a hybrid model, spending 40% of time working from home and 60% collaborating face to face in a hub. We offer competitive pay of up to £51,528, a generous civil service pension with employer contributions of 28.97%, and a leave package starting at 26 days rising to 31 days, plus up to 3 extra flexi days each month. Our benefits also include volunteering and charitable giving time, discounts and savings, interest-free loans for bikes or season tickets, and professional development, coaching, mentoring, and career progression opportunities. We have an award-winning, inclusive culture and have been recognised for diversity, leadership, and being a great place to work in digital.
last updated 36 week of 2026