About this role
Data Engineer
Must have below
- 10+ Years Experience Great Communicator/Client Facing/ Attention to detail Individual Contributor and ability to work as a team.
- 100% Hands on in the mentioned skills Programming Skills: Python and Data Pipelines: Advanced Proficiency in Python concepts like Code Structures, Modules, Packages, Class, SubClass, Inheritance, Multi-Threading and Functional Programming.
- Experience in developing reusable Python packages for internal or public usage Ability to write automating ETL processes and scheduling jobs like Airflow DAG.
- Ability to track job pipeline runs to reprocess error records Ability to orchestration different pipelines to run in sequence or parallel.
- Troubleshoot data pipeline errors and fix issues Export or Import data to/from various formats like CSV, JSON, XML etc preferably from S3 or other cloud storage.
- Experience in using AI IDE tool SQL: Advanced SQL skills, including complex joins, CTE's and subqueries Experience in optimizing SQL queries for performance and optimization in data warehouse technologies preferably Snowflake Testing and documentation: Proficiency in Python unit, integration and system test.
- Proficiency in implementing DBT tests for data validation and quality checks Code Generation: Experience in generating code using configurations using python and jinja templates Version control: Experience in GitHub, including implementing CI/CD process from scratch AWS Expertise: Data Storage solutions: In depth understanding of AWS S3 for data storage, ECS, IAM including best practices for organization and security Cloud Security: Knowledge of AWS security best practices, including IAM roles, encryption standard, secure coding guidelines, DBT profiles access configurations and more.
- Data Integration (nice to have): Experience with AWS lambda for serverless data processing tasks