About this role
Job Description Principal Databricks Data Engineer
Experience Required: 12?18 Years
Key Requirements
? 12?18 years of overall Data Engineering experience
? 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
? 5+ years of hands-on experience with Databricks and Apache Spark at scale
? Strong experience modernizing legacy Cloudera platforms including:
? CDH/CDP
? Hive
? HBase
? Impala
? Spark
? Modernize Cloudera platforms to Databricks Lakehouse architecture
? Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
? Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
? Ensure data reconciliation, audit integrity, and consistency during migration
? Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
? Implement layered data architecture including:
? Raw / Landing Layer
? Curated / Conformed Layer
? Semantic / Consumption Layer
? Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
? Strong experience with finance and risk data models including:
? General Ledger
? Sub-ledger
? Financial Hierarchies
? Credit Risk
? Liquidity Risk
? Market Risk
? Enable reporting capabilities including:
? Aggregation
? Drill-down
? Drill-back
? Build and manage semantic and consumption layers for BI, reporting, and analytics
? Define business metrics, dimensions, hierarchies, and KPIs
? Experience with:
? Databricks SQL
? Delta Tables
? dbt or similar frameworks
? Develop and optimize large-scale data pipelines using:
? PySpark
? Spark SQL
? Delta Lake
? Implement Medallion Architecture including:
? Bronze Layer
? Silver Layer
? Gold Layer
? Optimize workloads using:
? Z-ORDER
? OPTIMIZE
? Caching
? Cluster Configuration Tuning
? Implement:
? Data Governance
? Data Quality Frameworks
? Reconciliation Controls
? Exception Handling
? Establish data lineage and metadata management
? Ensure data security, access control, and compliance standards
? Experience with AWS or Azure cloud platforms
? Experience with CI/CD pipelines using:
? Git
? Terraform
? Jenkins
? Azure DevOps
? Familiarity with:
? Apache Airflow
? Databricks Workflows
? Experience with dbt is an advantage
? Act as a technical authority and lead enterprise architecture decisions
? Mentor senior engineers and establish engineering standards
? Collaborate with finance, risk, analytics, and governance stakeholders
? Translate complex data structures into business-ready insights
Nice-to-Have Skills
<d