Big Data Engineer

5 days ago

Apply Now

Receive Emails with Similar Jobs

Jobgether

WebsiteLinkedInAll Job Openings

11 - 50

💰 Seed Round on 2023-03

Description

•Jobgether has ALL remote jobs globally. We match you to roles where you're most likely to succeed and provide feedback on every application to help you learn. No more guesswork, application black holes, or recruiter ghosting in your job search. •In this role, you will work closely with cross-functional teams within the Business Analytics & Insights (BAI) department to deliver top-tier data solutions in domains such as Supply Chain, Finance, Operations, Customer Experience, HR, Risk Management, and Global IT. You will play a key part in the technical planning, execution, and maintenance of data migration processes. The role requires designing scalable, efficient data pipelines, ensuring data quality and integrity, and staying updated on the latest advancements in cloud and data engineering. •Accountabilities: •Lead the technical planning and execution of data migration, including ingestion, transformation, and storage within Azure Data Factory and Azure Data Lake. •Design and implement scalable data pipelines for seamless data movement from various sources using Azure Databricks. •Develop reusable frameworks for ingesting large datasets and implement data validation and cleansing mechanisms. •Work with real-time streaming technologies to process and ingest data effectively. •Provide technical support during and after migration, resolving challenges as they arise. •Stay up-to-date on advancements in cloud computing and data engineering, recommending best practices and industry standards.

Requirements

•8+ years of IT experience with at least 4 years working with Azure Databricks. •Strong proficiency in Python for data engineering, along with expertise in PySpark and SQL. •Knowledge of Azure Data components including Data Factory, Data Lake, SQL Data Warehouse (DW), and Azure SQL. •Experience in data modeling, source system analysis, and developing technical designs for data flows. •Familiarity with data profiling, cataloging, and mapping processes. •Experience with data visualization and exploration tools.