About this role
We are looking for a Data Engineer with strong Microsoft SQL Server and Snowflake experience to join our Data & Analytics engineering team. This role is ideal for an engineer who enjoys working with large-scale data, complex SQL, data pipelines, and production systems and wants to play a key role in modernizing a large SQL Server-based data platform to Snowflake and AWS. You will work across both our existing SQL Server environment and our modern cloud data platform, developing new capabilities while helping migrate and optimize existing data processing.
- Tech Stack Languages : SQL, Python, C#; scripting experience Databases : Microsoft SQL Server, Snowflake Data Engineering : ETL/ELT, Apache Airflow, data pipelines Cloud : AWS, including S3 and other AWS data services Development : Git, CI/CD, Liquibase, automated testing and deployment Data : Market data, trade data, transaction cost analysis, and financial data Develop, maintain, and optimize complex SQL Server queries, stored procedures, functions, views, and data processing workflows.
- Design and develop scalable Snowflake data solutions, including data models, SQL transformations, loading processes, and analytical workloads.
- Participate in the migration of large volumes of data and processing from SQL Server to Snowflake.
- Analyze existing SQL Server logic and translate or redesign it for Snowflake where appropriate.
- Build and maintain ETL/ELT pipelines using SQL, Python, and workflow orchestration tools such as Apache Airflow.
- Develop data ingestion and processing workflows using AWS, including S3.
- Perform SQL Server and Snowflake performance tuning, including query optimization, execution plans, indexing, statistics, warehouse sizing, and workload optimization.
- Build data validation and reconciliation processes to ensure migrated and transformed data is accurate and complete.
- Troubleshoot production data issues, identify root causes, and implement reliable long-term solutions.
- Implement monitoring, logging, alerting, and operational controls for critical data pipelines.
- Collaborate with software engineers, service managers and implementation team, product teams, and DevOps.
- Support and enhance existing Transaction Cost Analysis (TCA) products and develop new functionality.
- Work with large-scale trade, market, benchmark, and financial data.