About this role
Position Summary, Responsibilities and Expectations
- Design, develop, and optimize scalable ETL/ELT pipelines for enterprise data ingestion, integration, and transformation.
- Build and maintain data engineering workflows using Databricks, dbt, PySpark, SQL, and Python.
- Develop Data Lake, Data Warehouse, and Lakehouse solutions, including data modeling and incremental processing.
- Collaborate with business stakeholders to understand requirements, business processes, and challenges, and translate them into effective data and analytics solutions.
- Proactively communicate with business users and technical teams to clarify requirements, provide project updates, address issues, and ensure solutions meet business expectations.
- Ensure data quality, consistency, security, and governance through validation and automated testing.
- Prepare and integrate enterprise data to support BI, AI/ML applications, and advanced analytics.
- Develop AI/LLM solutions, including RAG, AI agents, API integrations, and intelligent workflow automation.
- Monitor and optimize data pipeline performance, reliability, resource utilization, and costs.
- Implement data orchestration, CI/CD, automated deployment, and monitoring practices.
- Collaborate with BI, AI, IT, and business teams to support solution delivery, user testing, documentation, and continuous improvement.
Essential Skills and Experience
- 2–4 years of hands-on experience in Data Engineering, ETL/ELT, and enterprise data platforms.
- Strong SQL and Python skills, with practical experience in Databricks, PySpark, Delta Lake, and dbt.
- Experience designing, developing, and optimizing production data pipelines, data models, and Lakehouse/Data Warehouse solutions.
- Ability to understand business processes, gather and clarify requirements, and translate business needs into actionable technical solutions.
- Strong verbal and written communication skills, including the ability to explain technical concepts to non-technical stakeholders.
- Strong analytical, problem-solving, and interpersonal skills, with a proactive attitude and ability to collaborate across business and technical teams.
- Familiarity with cloud data platforms (Azure/Microsoft Fabric preferred), Git, CI/CD, and data quality practices.
- Understanding of AI/LLM technologies, API integrations, and AI-driven data processing or automation.
- Nice-to-Have: Experience with RAG, AI agents, vector databases, Power BI, or enterprise ERP systems.