About this role
We are looking for a Data Engineer to be responsible for building and maintaining the infrastructure that supports the organization’s data architecture.
The role involves creating and managing data pipelines using Airflow for data extraction, processing, and loading, ensuring their maintenance, monitoring, and stability.
The engineer will work closely with data analysts and end-users to provide accessible and reliable data
Main Tasks and Responsibilities:
Responsible for maintaining the infrastructure that supports the current data architectureResponsible for creating data pipelines in Airflow for data extracting, processing, and loadingResponsible for data pipelines maintenance, monitoring, and stabilityResponsible for providing data access to data analysts and end-usersResponsible for DevOps infrastructureResponsible for deploying Airflow dags to production environment using DevOps toolsResponsible for code and query optimizationResponsible for code reviewResponsible for improving the current data architecture and DevOps processesResponsible for delivering data in useful and appealing ways to usersResponsible for performing and documenting analysis, review, and study on specified regulatory topicsResponsible for understanding business change and requirement needs, assess the impact and the cost Technical skills:
Advanced Python(Mandatory)Experience in creating APIs in Python - At least Flask (Mandatory)Experience in documenting and testing in Python (Mandatory)Advanced SQL skills and relational database management (Oracle is Mandatory, SQL server is desirable, PostgreSQL is desirable )Experience with Data WarehousesHadoop ecosystem - HDFS + Yarn (Mandatory)Spark Environment Architecture (Mandatory)Advanced PySpark - (Mandatory)Experience in creating and maintaining distributed environments using Hadoop and SparkData Lakes - Experience in organizing and maintaining data lakes (Mandatory) - S3 is preferredExperience with Parquet file format is mandatory, Avro is a plusApache Airflow - Experience in both pipeline development and deploying Airflow in distributed environment (Mandatory)Containerization - Docker is MandatoryKubernetes (Mandatory)Apache Kafka (Mandatory)Experience in automating applications deployment using DevOps tools - Jenkins is Mandatory, Ansible is a plusAgile methodologies (At least SCRUM)
Language Skills:
Fluent in English (Mandatory)