Data Engineer

2 days ago

Apply Now

Receive Emails with Similar Jobs

PeerIslands

WebsiteLinkedInAll Job Openings

High velocity digital transformation

Azure • AWS • Data Engineering • Cloud Native Applications • Cloud Transformation

11 - 50

Description

• We are seeking a highly skilled and experienced Data Engineer to join our dynamic team. • This role focuses on leveraging Apache Spark and Scala/Python for building sophisticated big data solutions. • The ideal candidate will be adept at designing and implementing robust, scalable data processing systems that drive our business insights and decision-making. • Develop and maintain scalable and reliable data architectures. • Ensure seamless data flow from various sources to our storage and analysis platforms. • Monitor and optimize the performance of data processing applications. • Ensure the accuracy and consistency of data through rigorous testing and validation processes. • Work closely with analysts, and IT teams to understand data needs. • Continuously update technical knowledge and skills.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. • Minimum of 3 years Proven experience in data engineering, specifically with Apache Spark and Scala/Python • Familiarity with cloud platforms (Azure, AWS, GCP) and their data services is a plus. • Hands-on experience in Databricks with Pyspark and Unity Catalog • Experience with dataflow and Data factory pipelines. • Practical experience in handling error or invalid formats in incoming data. • Experience working with Delta table formats. • Excellent problem-solving skills and ability to optimize data systems for performance and scalability. • Generative AI and machine learning concepts, with experience in integrating ML models into big data platforms will be a plus • Strong communication skills for effective collaboration within and across teams.