About this role
Job Description: Responsibilities:
• Develop & Optimize Data Pipelines • Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark). • Optimize performance and cost-efficiency of Spark jobs. • Ensure data quality through validation, monitoring, and alerting mechanisms. • Understand cluster types, configuration, and use-case for serverless
• Implement Unity Catalog for Data Governance • Design and enforce access control policies using Unity Catalog. • Manage data lineage, auditing, and metadata governance. • Enable secure data sharing across teams and external stakeholders.
• Integrate with Cloud Data Platforms • Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams. • Implement Delta Lake for scalable, ACID-compliant storage.
• Automate & Orchestrate Workflows • Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory. • Monitor and troubleshoot failures in job execution and cluster performance.
• Collaborate with Stakeholders • Work with Data Analysts, Scientists, and Business Teams to understand requirements. • Translate business needs into scalable data engineering solutions.
• API expertise • Ability to pull data from a wide variety of APIs using different strategies and methods
Required Skills & Experience:
• Azure Databricks & Apache Spark (PySpark) – Strong experience in building distributed data pipelines. • Python – Proficiency in writing optimized and maintainable Python code for data engineering. • Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking. • SQL – Strong knowledge of SQL for data transformations and optimizations. • Delta Lake – Understanding of time travel, schema evolution, and performance tuning. • Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory. • CI/CD & Infrastructure as Code (IaC) – Familiarity with Databricks CLI, Databricks DABs, and DevOps principles. • Security & Compliance – Knowledge of IAM, role-based access control (RBAC), and encryption.
Preferred Qualifications:
• Experience with MLflow for model tracking & deployment in Databricks. • Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid). • Hands-on experience with dbt (Data Build Tool) for modular ETL development. • Certification in Databricks, Azure is a plus. • Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server • Experience with Azure Synapse Link for Dynamics, dataverse • Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc
Soft Skills:
• Strong problem-solving and debugging skills. • Ability to work independently and in teams. • Excellent communication and documentation skills.