Data Engineer - Spark

July 11

Apply Now

Receive Emails with Similar Jobs

Factored

WebsiteLinkedInAll Job Openings

Machine Learning • Data Engineering • Data Analytics • Artificial Intelligence • Data Science

51 - 200

Description

• Responsible for planning, implementing, directing, and administering all aspects of the human resources function for the Mid-Atlantic Region • Provide day-to-day management of the Employee Relations team

Requirements

• Proven experience in developing and managing real-time and batch data infrastructures. • Strong proficiency in SQL, Python, and Spark. • Experience across the entire software development lifecycle, from inception to production and monitoring. • Demonstrated ability to ensure data quality and accuracy in production environments. • Familiarity with cloud services such as AWS, GCP or Azure. • Excellent problem-solving skills and the ability to lead projects from conception to implementation.

Benefits

• Develop, enhance, and manage both real-time and batch data infrastructures. • Apply your expertise in SQL, Python, and Spark, alongside technologies such as HDFS, Snowflake, Hive, HBase, Scylla, Django, and FastAPI. • Implement processes to guarantee the highest standards of data quality and accuracy across all production data systems. • Collaborate with Data Engineers to optimize data models and workflows, ensuring efficiency and scalability. • Develop robust ETL processes for comprehensive analysis and reporting to support data analytics processes. • Partner with Product Managers to design and build innovative data products that drive business value and cooperate with the DevOps team to scale and optimize data infrastructure, ensuring reliability and performance. • Participate in architecture discussions, influence the technology roadmap, and take ownership of new projects and initiatives.