About this role
Job Description Description:
• Designing, developing, and optimizing Azure Data Factory and Databricks pipelines from Oracle databases to Lakehouses
• Designing and optimizing data models using Medallion Architecture principles
• Designing, developing and optimizing data connections from Databricks Medallion Architecture to on-premises data sources for downstream consumers.
• Designing and optimizing relational data models
• Translating Informatica ETL to Azure Data Factory and Databricks ELT
Knowledge Transfer:
• Transfer From DataOps/Cloud Data Engineer to Designated CSC Resource
When Knowledge Will Be Transferred:
• Knowledge transfer must be completed one week prior to the end of the project or one week prior to the consultant leaving the ministry.
What Knowledge Will Be Transferred:
• All deliverables, including design/supporting/release/training documents must be checked into designated version control repositories (for example, git repositories or SharePoint). All final documents and working drafts related to project requirements or solution design must be stored on designated project repositories (for example, SharePoint site, HPQC, TFS, etc.)
• Project manager and designated ministry staff must be regularly informed in writing (by email) of where documentation has been stored and must be provided a minimum of one walk-through of all documentation as part of the final knowledge transfer activities.
How Knowledge Will Be Transferred:
• Knowledge will be transferred through 1 on 1 sessions, emails, document updates and document review with the team.
Requirements Experience and Skill Set Requirements:
Must Haves:
• Experience developing ETL processes with Informatica, Azure Data Factory and Databricks (Python, Lakeflow, and SQL Optimization)
• Experience with solution development with Databricks in Medallion Architecture.
• Experience with Cloud data engineer must be familiar and experienced with