Big Data Engineer

August 3

Apply Now

Receive Emails with Similar Jobs

MGID

WebsiteLinkedInAll Job Openings

Global Advertising Platform

Native advertising • paid content distribution • audience development • website traffic management • advertising

501 - 1000

💰 Venture Round on 2009-11

Description

• Collaborate with Data Scientists, Data Analysts, and other stakeholders to understand data needs and develop solutions. • Design, develop, and optimize PySpark applications for processing and analyzing large sets of structured and unstructured data. • Monitor and evaluate data to ensure accuracy and integrity, troubleshoot and debug PySpark code. • Build and maintain data pipelines for ingesting, processing, and storing data, optimizing for performance and scalability. • Develop and maintain data visualization dashboards and reports to enable insights and decision-making. • Create and maintain tools and libraries for efficient data processing. • Stay up-to-date with industry trends and new technologies to continuously improve data processing capabilities.

Requirements

• Proven experience in developing and optimizing PySpark applications. • Strong knowledge of distributed computing principles and concepts. • Practical experience working with large datasets using technologies such as Hadoop, Spark, ClickHouse. • Proficiency in programming languages such as Python, SQL. • Experience with Linux/Unix command-line interface. • Familiarity with data visualization and dashboarding tools. • Strong communication skills and ability to work effectively in a remote team environment. • Excellent problem-solving skills and attention to detail. • Will be a plus: Bachelor's or Master's degree in Computer Science or a related field. • Practical experience with ClickHouse. • Practical experience with stream processing and messaging systems such as Kafka. • Practical experience with NoSQL databases (for example MongoDB), especially Aerospike. • Knowledge of AdTech domain - understanding of online advertising, RTB. • Familiarity with containerization technologies such as Docker and Kubernetes, cloud computing platforms. • Familiarity with data governance and security best practices. • Knowledge of machine learning concepts and frameworks.