Data Platform Engineer
March 6, 2023
Description
• Contribute directly to Apache Hudi and the surrounding open source ecosystem. • Be the thought leader around all things data engineering within the company. • Implement new sources and connectors to seamlessly ingest data streams. • Building scalable job management on Kubernetes to ingest, store, manage and optimize petabytes of data on cloud storage. • Optimize Spark or Flink applications to flexibly run in batch or streaming modes based on user needs, optimize latency vs throughput. • Tune clusters for resource efficiency and reliability, to keep costs low, while still meeting SLAs.
Requirements
• 3+ years of experience in building and operating data pipelines in Apache Spark or Apache Flink • 2+ years of experience with workflow orchestration tools like Apache Airflow, Dagster • Proficient in Java, Maven, Gradle and other build and packaging tools • Adept at writing efficient SQL queries and trouble shooting query plans • Experience managing large-scale data on cloud storage • Great problem-solving skills and eye for details • Operational excellence in monitoring, deploying, and testing job workflows • Open-minded, collaborative, self-starter, fast-mover
Benefits
• Unlimited PTO • Paid Parental Leave • Equity • Flexible Schedule
Apply Now