Big Data Integrations Engineer

September 16

Apply Now

Receive Emails with Similar Jobs

MI-C3 International Ltd

WebsiteLinkedInAll Job Openings

Operations Management • Decision Suppost Systems • Task Management • Asset Management • IoT Products

51 - 200

Description

• As a Data Integration Engineer, you will design, implement, and maintain data integration solutions for real-time streaming. • Work with a diverse range of Big Data tools and technologies. • Collaborate with cross-functional teams to understand data requirements, source systems, and data formats. • Develop custom data processing components and applications using Java and Python. • Implement ETL processes to ingest and transform data from various streaming sources. • Optimize data processing workflows to ensure timely handling of streaming data. • Utilize and maintain various Big Data tools such as Apache NiFi, Spark, Kafka, Druid/TimescaleDB. • Set up message brokers like RabbitMQ, AMQP, and Kafka for efficient data exchange. • Integrate IoT/IIoT protocols to capture data from edge devices and industrial systems. • Implement data validation checks and data quality measures. • Monitor performance of data integration pipelines and troubleshoot issues.

Requirements

• Strong experience in designing and implementing data integration solutions for real-time streaming data. • Proficiency in using Big Data technologies such as Apache NiFi, Apache Spark, Kafka, and Druid. • Familiarity with message brokers like RabbitMQ, AMQP, and Kafka for data exchange and event-driven architectures. • Hands-on experience with IoT/IIoT protocols such as MQTT, SNMP, CoAP, TCP, and WebSockets. • Proficiency in programming languages such as Java and Python for developing custom data processing components. • Understanding of data streaming platforms and experience with Rust is a plus. • Knowledge of data quality assurance and validation techniques to ensure reliable data. • Ability to troubleshoot and resolve issues related to data integration and streaming processes. • Strong analytical and problem-solving skills, with a keen eye for detail. • Excellent communication and teamwork skills to collaborate effectively with cross-functional teams. • Experience with cloud-based platforms and distributed systems is advantageous. • Have a never ending curious mindset to learn and work with new tools and technologies.