About this role
We are hiring a Data Engineer to build and run scalable batch data pipelines on our big data platform. You will write production-grade Python, PySpark and SQL, orchestrate workflows in Airflow, and work with data stored in Hadoop and Hive. You will partner with analysts, data scientists and product teams to deliver reliable, well-modelled data that the business can trust.
Key Responsibilities
- Design, build and maintain scalable ETL/ELT pipelines using Python, PySpark and SQL.
- Develop and schedule workflows in Apache Airflow (DAG design, dependencies, retries, SLAs, alerting).
- Ingest, transform and store large datasets on Hadoop (HDFS) and expose them through Hive tables.
- Design Hive schemas with sound partitioning, bucketing and file-format choices (Parquet/ORC).
- Tune PySpark jobs and SQL queries for performance and cost (partitioning, joins, caching, skew handling).
- Build data quality checks, validation, monitoring and reconciliation into every pipeline.
- Troubleshoot production failures, perform root-cause analysis and support on-call or incident rotations.
- Write clean, tested, version-controlled code and take part in code reviews.
- Document data flows, data models and runbooks.
- Collaborate with analysts, data scientists and business stakeholders to translate requirements into data solutions.
- Bachelor's degree in Computer Science, Engineering, IT or a related field.
- Experience with Git and CI/CD pipelines.
- Exposure to a cloud platform (AWS, Azure or GCP) and its data services.
- Knowledge of data modelling (star schema, dimensional modelling) and data warehousing concepts.
- Familiarity with Kafka or other streaming tools, and with Linux shell scripting.
- Experience with Docker or Kubernetes.
- Understanding of data governance, security and lineage practices.
- Experience working in Agile/Scrum teams.