About this role
How you'll make an impact
• Design and build production pipelines on Databricks using Spark Declarative Pipelines (SDP) and PySpark, from raw ingestion through business-ready data products. • Define all pipelines, jobs, and schedules as code in Databricks Asset Bundles, deployed to every environment through automated CI/CD. • Build data quality, monitoring, and lineage into pipelines so issues are caught and diagnosed before they reach consumers. • Turn recurring solutions into reusable frameworks, standards, and shared libraries that raise delivery speed across the team. • Own your pipelines in production — performance, cost, reliability, and incident response. • Partner with Digital Product Managers, architects, and business stakeholders to translate requirements into technical designs, and mentor engineers newer to the platform. What you'll need (Required)
• Bachelor's degree in computer science, engineering, or a related technical field, plus five or more years of data engineering experience, including hands-on production experience on Databricks. • Demonstrated experience with Spark Declarative Pipelines (SDP / Delta Live Tables) — streaming tables, materialized views, expectations, Auto Loader, and CDC patterns. • Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments. • Strong Spark and PySpark skills, including performance tuning, and production-quality Python beyond notebook scripting. • Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL. • Experience with Git-based CI/CD in a shared repository — code review, automated validation, and promotion across environments. • Demonstrated ability to take ambiguous requirements through design to production independently, and to make and defend sound technical decisions. • "Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication." • "Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads." What else we look for (Preferred)
• Hands-on experience with Spark Declarative Pipelines (SDP / Delta Live Tables), including streaming tables, materialized views, expectations, Auto Loader, and CDC patterns. • Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments. • Strong Spark and PySpark development skills, including performance tuning, and production-quality Python beyond notebook scripting. • Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL. • Experience with Git-based CI/CD for data platforms, including code review, automated validation, and promotion across environments. • Experience with cloud data platforms on AWS, metadata-driven ingestion frameworks, and infrastructure-as-code. • Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication. • Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads. • Familiarity with the broader modern data ecosystem, such as Snowflake, dbt, Kafka, and Airflow. • Experience integrating enterprise and clinical source systems, such as Epic, SAP, or Salesforce, or migrating workloads from legacy ETL platforms onto a lakehouse. • Practical understanding of governance, quality, security, validation, and support expectations in a regulated enterprise environment. • Ability to provide technical guidance, coach team members, and contribute to reusable standards, documentation, and delivery practices.