Senior Manager @ EXL

EXL (NASDAQ: EXLS) is a leading data analytics and digital operations and solutions company. We partner with clients using a data and AI-led approach to reinvent business models, drive better business outcomes and unlock growth with speed. EXL harnesses the power of data, analytics, AI, and deep industry knowledge to transform operations for the world’s leading corporations in industries including insurance, healthcare, banking and financial services, media and retail, among others. EXL was founded in 1999 with the core values of innovation, collaboration, excellence, integrity and respect. We are headquartered in New York and have more than 54,000 employees spanning six continents. For more information, visit www.exlservice.com.

EXL never requires or asks for fees/payments or credit card or bank details during any phase of the recruitment or hiring process and has not authorized any agencies or partners to collect any fee or payment from prospective candidates. EXL will only extend a job offer after a candidate has gone through a formal interview process with members of EXL’s Human Resources team, as well as our hiring managers.

Role: Databricks Engineer

Experience: 9-12 Years

Location: ALL EXL Locations

Work Mode: Hybrid

Key Role and Responsibilities:

 Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production environments.

 Configure and manage Unity Catalog for data governance, fine-grained access control, permissions, metadata management, and data lineage.

 Optimize Databricks cluster configurations, including instance types, auto-scaling, spot/preemptible nodes, and compute pools to improve performance and reduce costs.

 Implement workspace best practices, including folder structures, access controls, secret management using Databricks Secrets, Azure Key Vault, or AWS Secrets Manager.

 Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job orchestration with dependency management.

 Design and implement Delta Lake tables using partitioning, Z-Ordering, OPTIMIZE, VACUUM, and file compaction techniques.

 Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers) for scalable and governed data lakehouse solutions.

 Develop Delta Live Tables (DLT) pipelines with built-in data quality expectations for reliable ETL/ELT processing.

 Manage schema evolution, table versioning, Time Travel, and Change Data Feed (CDF) to support incremental data processing.

 Design and implement lakehouse architectures integrating Delta Lake with cloud storage and external systems such as Azure Data Lake Storage (ADLS), Kafka, Event Hubs, and Kinesis.

 Develop scalable batch and real-time data pipelines using PySpark, Spark SQL, Structured Streaming, and Delta Lake.

 Build streaming ingestion pipelines from Kafka, Azure Event Hubs, and other streaming platforms into Delta tables.

 Optimize PySpark applications using broadcast joins, Adaptive Query Execution (AQE), dynamic partition pruning, caching, and Photon Engine.

 Develop reusable transformation frameworks, utility libraries, and pipeline templates to improve engineering productivity and standardization.

 Implement robust error handling, retry mechanisms, logging, monitoring, and dead-letter queue (DLQ) patterns for production-grade pipelines.

 Set up and manage MLflow experiment tracking, model registry, and model lifecycle management.

 Support machine learning workloads by enabling scalable model training, inference, and GPU-based compute environments.

 Develop feature engineering pipelines using Databricks Feature Store to create reusable and versioned machine learning features.

 Enable Generative AI solutions, including Retrieval-Augmented Generation (RAG), vector search, LLM fine-tuning, and Mosaic AI capabilities.

 Implement MLOps best practices, including model versioning, model deployment, A/B testing, and Databricks Model Serving.

 Integrate Databricks with Azure Data Lake Storage (ADLS) and other cloud-native services.

 Develop and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or GitLab CI for Databricks notebooks, jobs, and workflows.

 Automate Databricks infrastructure deployment using Databricks Asset Bundles (DABs), Terraform, and Infrastructure-as-Code (IaC) practices.

 Build and manage data ingestion frameworks using Auto Loader, COPY INTO, and third party integration tools such as Fivetran, dbt, and Airbyte.

 Monitor pipeline execution, cluster utilization, system performance, and cloud costs using Databricks system tables and cloud monitoring tools.

 Implement row-level security, column-level masking, dynamic views, and governance policies using Unity Catalog.

 Enforce data quality through Delta Live Tables expectations and Great Expectations frameworks.

 Perform query optimization, execution plan analysis, caching strategies, and performance tuning to improve workload efficiency.

 Maintain enterprise data cataloging, metadata management, and end-to-end data lineage.

 Prepare technical documentation, architecture diagrams, operational runbooks, and standard operating procedures for Databricks platform and data engineering solutions.

Qualifications for Candidates

 Bachelor’s or master’s degree in computer science, Information Technology, Data Engineering, or related field.

 4+ years of total experience in data engineering or software engineering.

 3+ years of dedicated hands-on experience with the Databricks platform in production environments.

 Strong background in big data engineering, cloud data platforms, and distributed computing.

 Deep expertise in Databricks Workspaces, Clusters, Jobs, Workflows, and Repos.

 Proficiency with Unity Catalog — metastore setup, catalog/schema/table management, access controls, and data lineage.

 Hands-on experience with Delta Live Tables (DLT) — pipeline development, expectations, and monitoring.

 Strong command of Delta Lake internals — transaction log, ACID guarantees, file layout, and optimization techniques.

 Experience with Databricks SQL Warehouses, SQL Analytics, and dashboard creation.

 Knowledge of Databricks Photon engine, serverless compute, and cost optimization strategies.

 4+ years of PySpark development — Dataframe, Datasets, Spark SQL, RDD operations.

 Expert-level SQL — window functions, lateral joins, CTEs, recursive queries, and analytical functions.

 Experience with Spark performance tuning — AQE, query plans (EXPLAIN), partitioning, and caching.

 Proficiency with Python for pipeline development, utilities, and automation.

 Hands-on experience with at least one: Azure (ADLS Gen2, ADF, Azure Databricks), AWS (S3, EMR, Glue, AWS Databricks), or GCP (GCS, BigQuery, Dataproc).  Experience with cloud networking for Databricks: VNet/VPC injection, private endpoints, and firewall configurations.

 Familiarity with IAM roles, managed identities, and service principal authentication for Databricks.

(Nice to Have):

 Working knowledge of MLflow — experiment tracking, model registry, and deployment.

 Experience supporting ML pipelines on Databricks for training, evaluation, and serving.

Education: UG - Any

 Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production environments.  Configure and manage Unity Catalog for data governance, fine-grained access control, permissions, metadata management, and data lineage.  Optimize Databricks cluster configurations, including instance types, auto-scaling, spot/preemptible nodes, and compute pools to improve performance and reduce costs.  Implement workspace best practices, including folder structures, access controls, secret management using Databricks Secrets, Azure Key Vault, or AWS Secrets Manager.  Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job orchestration with dependency management.  Design and implement Delta Lake tables using partitioning, Z-Ordering, OPTIMIZE, VACUUM, and file compaction techniques.  Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers) for scalable and governed data lakehouse solutions.  Develop Delta Live Tables (DLT) pipelines with built-in data quality expectations for reliable ETL/ELT processing.  Manage schema evolution, table versioning, Time Travel, and Change Data Feed (CDF) to support incremental data processing.  Design and implement lakehouse architectures integrating Delta Lake with cloud storage and external systems such as Azure Data Lake Storage (ADLS), Kafka, Event Hubs, and Kinesis.  Develop scalable batch and real-time data pipelines using PySpark, Spark SQL, Structured Streaming, and Delta Lake.  Build streaming ingestion pipelines from Kafka, Azure Event Hubs, and other streaming platforms into Delta tables.  Optimize PySpark applications using broadcast joins, Adaptive Query Execution (AQE), dynamic partition pruning, caching, and Photon Engine.  Develop reusable transformation frameworks, utility libraries, and pipeline templates to improve engineering productivity and standardization.  Implement robust error handling, retry mechanisms, logging, monitoring, and dead-letter queue (DLQ) patterns for production-grade pipelines.  Set up and manage MLflow experiment tracking, model registry, and model lifecycle management.  Support machine learning workloads by enabling scalable model training, inference, and GPU-based compute environments.  Develop feature engineering pipelines using Databricks Feature Store to create reusable and versioned machine learning features.  Enable Generative AI solutions, including Retrieval-Augmented Generation (RAG), vector search, LLM fine-tuning, and Mosaic AI capabilities.  Implement MLOps best practices, including model versioning, model deployment, A/B testing, and Databricks Model Serving.  Integrate Databricks with Azure Data Lake Storage (ADLS) and other cloud-native services.  Develop and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or GitLab CI for Databricks notebooks, jobs, and workflows.  Automate Databricks infrastructure deployment using Databricks Asset Bundles (DABs), Terraform, and Infrastructure-as-Code (IaC) practices.  Build and manage data ingestion frameworks using Auto Loader, COPY INTO, and third party integration tools such as Fivetran, dbt, and Airbyte.  Monitor pipeline execution, cluster utilization, system performance, and cloud costs using Databricks system tables and cloud monitoring tools.  Implement row-level security, column-level masking, dynamic views, and governance policies using Unity Catalog.  Enforce data quality through Delta Live Tables expectations and Great Expectations frameworks.  Perform query optimization, execution plan analysis, caching strategies, and performance tuning to improve workload efficiency.  Maintain enterprise data cataloging, metadata management, and end-to-end data lineage.  Prepare technical documentation, architecture diagrams, operational runbooks, and standard operating procedures for Databricks platform and data engineering solutions.

Graduate in Computer Science, Data Science, or related field. 9-12 years of experience in data engineering or related field.