Natobotics
IT Consulting, IT Architecture • Financial Services, Health care, Media, Telecom • Big Data - Storm , Java, Oracle • Business Analysis, Development and Testing services
51 - 200
Azure Data Engineering Lead
November 26
Natobotics
IT Consulting, IT Architecture • Financial Services, Health care, Media, Telecom • Big Data - Storm , Java, Oracle • Business Analysis, Development and Testing services
51 - 200
Description
• Microsoft Azure • Experience developing and deploying data intensive applications at enterprise scale. • Evolve architecture to support scaling to other OUs and bottlers. • Topics include data privacy, clean team, data security, ML scaling and performance. • Provide technical leadership and hands-on oversight in developing data processing applications on Spark using Scala programming, focusing on Microsoft Azure Synapse Spark Runtime. • Design and optimize data pipelines processing through various zones in Medallion architecture using Azure Synapse pipelines. • Manage data ingestion, ensure data quality checks with tools like Deequ, and handle data validation and error management. • Develop and manage configuration settings using JSON files read by classes such as ApplicationConfig and TableConfig for various zones. • Collaborate with data scientists, analysts, and cross-functional teams to ensure seamless integration and alignment of data engineering practices with marketing strategies. • Oversee logging, auditing, and error handling processes to track and ensure data processing integrity. • Knowledge of Azure Log Analytics and KQL queries a plus. • Implement unit testing with tools like Scala Test and maintain data quality checks for reliable data processing outcomes.
Requirements
• Minimum years of experience: 10 – 15 years
Apply Now