Staff Data Engineer - Infrastructure

September 20

Apply Now

Receive Emails with Similar Jobs

ASAPP

WebsiteLinkedInAll Job Openings

Artificial Intelligence • Contact Center • Customer Experience • Digital Transformation • Call Centers

201 - 500

Description

• Design and deploy improvements to our mission-critical production data pipeline, data warehouses, data systems • Recognize data flow patterns and generalizations to automate as much as possible to drive productivity gains • Expand our logging and monitoring processes to discover and resolve anomalies and issues before they become problems • Develop state-of-the-art automation and data solutions in Python, Spark and Flink • Maintain, Manage, Monitor our infrastructure related including Kafka, Kubernetes, Spark, Flink, Jenkins, general OLAP and RDBMS databases, S3 objects buckets, permissions • Increase the efficiency, accuracy, and repeatability of our ETL processes • Know how to make the tradeoffs required to ship without compromising quality

Requirements

• 12+ years of experience in general software development and/or dev-ops, sre roles in AWS. • 5+ years experience in data engineering, data systems, pipeline and stream processing. • Expertise in at least one flavor of SQL, e.g. Redshift, Postgres, MySQL, Presto/Trino, Spark SQL, Hive • Proficiency in a high-level programming language(s). We use Python, Scala, Java, Kotlin, and Go • Experience with CI/CD (continuous integration and deployment) • Experience with workflow management systems such as Airflow, Oozie, Luigi, and Azkaban • Experience implementing data governance, i.e. access management policies, data retention, IAM, etc. • Confidence operating in a devops-like capacity working with AWS, Kubernetes, Jenkins, Terraform, etc. thinking about automation, alerting, monitoring, and security and other declarative infrastructure