About this role
• Bachelor’s or higher degree in Computer Science or a related discipline; or equivalent (minimum 7 years work experience). • At least 4+ years of consulting or client service delivery experience on Azure • At least 4+ years of experience in developing data ingestion, data processing and analytical pipelines for big data, relational databases such as SQL server and data warehouse solutions such as Azure Synapse • Extensive experience providing practical direction with using Azure Native services. • Extensive hands-on experience implementing data ingestion, ETL and data processing using Azure services: ADLS, Azure Data Factory, Azure Functions, Azure Logic App Synapse/DW, Azure SQL DB, Event Hub, IOT Hub, Azure Stream Analytics, Azure Analysis Service, Databricks, Cosmo DB etc. • Minimum of 4+ years of hands-on experience in Azure and Big Data technologies such as Java, Python, SQL, ADLS/Blob, PySpark and SparkSQL, Databricks, HD Insight and live streaming technologies such as EventHub, Azure Stream Analytics etc. • Well versed in DevSecOps and CI/CD deployments • Cloud migration methodologies and processes including tools like Azure Data Factory, Data Migration Service, etc. • Minimum of 4+ years of RDBMS experience • Experience in using Big Data File Formats and compression techniques. • Experience working with Developer tools such as Azure DevOps, Visual Studio Team Server, Git, Jenkins, etc. • Experience with private and public cloud architectures, pros/cons, and migration considerations.
A Databricks Data Engineer is responsible for designing, building, and maintaining data solutions that leverage Databricks' powerful capabilities on the cloud platform. Key responsibilities include:
• Designing and deploying data pipelines that efficiently process vast amounts of data using Databricks, optimizing for performance and scalability. • Collaborating with data scientists and business stakeholders to understand requirements and provide data solutions tailored to those needs. • Implementing robust data models and architectures utilizing Databricks, ensuring data quality and integrity. • Monitoring and troubleshooting data pipelines, proactively identifying and resolving issues to ensure smooth data flows. • Ensuring data security, governance, and privacy measures are in place, adhering to industry best practices. • Optimizing data storage and processing costs while maintaining high performance. • Engaging in continuous improvement activities, researching new technologies and methodologies to enhance data solutions.
• Bachelor’s or higher degree in Computer Science or a related discipline; or equivalent (minimum 7 years work experience). • At least 4+ years of consulting or client service delivery experience on Azure • At least 4+ years of experience in developing data ingestion, data processing and analytical pipelines for big data, relational databases such as SQL server and data warehouse solutions such as Azure Synapse • Extensive experience providing practical direction with using Azure Native services. • Extensive hands-on experience implementing data ingestion, ETL and data processing using Azure services: ADLS, Azure Data Factory, Azure Functions, Azure Logic App Synapse/DW, Azure SQL DB, Event Hub, IOT Hub, Azure Stream Analytics, Azure Analysis Service, Databricks, Cosmo DB etc. • Minimum of 4+ years of hands-on experience in Azure and Big Data technologies such as Java, Python, SQL, ADLS/Blob, PySpark and SparkSQL, Databricks, HD Insight and live streaming technologies such as EventHub, Azure Stream Analytics etc. • Well versed in DevSecOps and CI/CD deployments • Cloud migration methodologies and processes including tools like Azure Data Factory, Data Migration Service, etc. • Minimum of 4+ years of RDBMS experience • Experience in using Big Data File Formats and compression techniques. • Experience working with Developer tools such as Azure DevOps, Visual Studio Team Server, Git, Jenkins, etc. • Experience with private and public cloud architectures, pros/cons, and migration considerations.