Senior Data Engineer - Databricks Platform
October 24
Description
• This is a hands-on Data Engineering role that will require strong Python/PySpark coding skills. • Contribute to the enhancements and continuous improvement of performance and reliability of the existing Data Platform Services to meet the requirements from Data Engineering/Product team • Design and build self servicing onboarding and automation capabilities in Core Data Platform services in a scalable, reliable and secure way. • Design and build platform capabilities to support onboarding and operation effectiveness • Design and develop scalable data platform and integration solutions for various data sources leveraging Databricks Unified Platform • Ensure data quality and integrity across all data systems. • Ensure data security and compliance with relevant regulations. • Implement best practices for data management and governance. • Develop and maintain documentation for data systems and processes. • Monitor and troubleshoot data platform issues. • Participate in on-call rotations/Pagerduty for data platform support.
Requirements
• Good understanding of Data Lakes, Data Warehouses is MUST to have • Software development, coding expertise in the Data engineering field using any of the Python/PySpark is MUST to have • Hands on experience using version control systems such as Git and CI/CD workflows and practice • Design and Develop data ingestion services that are highly performant, reliable and scalable • Decent expertise using ANSI SQL and Spark SQL is key to have • Workflow automation, orchestration using Airflow or equivalent tools stack • Hands on working knowledge on at least in one of these: Databricks, Hadoop and related stacks • Working experience in at least one of cloud services from any of the Amazon AWS, Google GCP or Microsoft Azure, preferably AWS • Streaming (Kafka) and batch based data sources diverse data sources and data formats (xml, json, yaml, parquet, avro, delta) and respective use cases • Any visualization experience is an advantage so you can bring clarity to prod incidents by using them is a nice to have • An excellent understanding of the nuances that add complexities around time zones, geo, various data formats, data types across different storage systems is nice to have • Agile development methodologies using the Atlassian suite: Jira, Confluence • Collaborate with cross-functional teams to deliver data solutions and provide technical support and guidance to team members. • Stay up-to-date with the latest data engineering technologies and trends.
Benefits
• Medical, Dental and Vision benefits for you and your family, including Flexible Spending Accounts (FSA) and Health Savings Accounts (HSAs) • Paid time off policy including holidays, sick time, and day off for your birthday • Free concert tickets • 401(k) program with company match, Stock Program Reimbursement • New parent programs & support including caregiver leave and baby bonus, infertility support • Tuition reimbursement, student loan repayment internal growth and development programs & trainings • Volunteer time off, crowdfunding network
Apply NowSimilar Jobs
October 24
Senior Data Engineer for automated data lake solution supporting analytics.
🇺🇸 United States – Remote
💵 $92.1k - $148.8k / year
🔥 Funding within the last year
💰 Private Equity Round on 2024-07
⏰ Full Time
🟠 Senior
🚰 Data Engineer
🦅 H1B Visa Sponsor
October 22
Join SDG as a Data Engineer to work on Cupid Media product.
October 20
Senior Data Engineer at Bitstrapped, working with Google Cloud and data solutions.
October 20
Design and maintain data architecture for MDCalc, a medical reference platform.
October 19
J.D. Power
501 - 1000
Senior associate in data engineering for automotive industry at J.D. Power.