Senior Data Engineer

June 22

Apply Now

Receive Emails with Similar Jobs

Pierce Professional Resources

WebsiteLinkedInAll Job Openings

Temporary Staffing • Contract Staffing • Permanent Staffing • Finance & Accounting Staffing • IT Staffing & Recruiting

11 - 50

Description

• Strategy Creation: Collaborate with cross-functional teams to define the data engineering strategy aligned to business objectives, including data modeling that unifies data assets across a range of source systems used to manage the operations of our partnering hospitals. • Pipeline Development: Define and execute processes needed to develop, test, deploy, and maintain high quality data pipelines. Oversee the end-to-end development of data pipelines from source data extraction through to production-grade analytical dataset delivery, ensuring data quality and security throughout the pipeline. • Performance Optimization: Continuously monitor and optimize data processing performance and efficiency. Identify and address bottlenecks, optimize query performance, and improve overall system stability. • Data Governance: Establish and enforce data quality management policies, data access controls, and data privacy standards. • Technical Leadership: Stay abreast of the latest developments in engineering tools and best practices. Provide guidance to the team about technical challenges. • Documentation: Maintain clear and comprehensive documentation of data pipelines, architecture, and processes to ensure knowledge sharing and team continuity. • Third-party Management: Evaluate and manage relationships with third-party vendors and tools, making informed decisions about when to leverage external solutions.

Requirements

• Experience with the Azure cloud ecosystem • Experience developing production-ready, real-time machine learning model serving pipelines • Comfort developing in the Apache Spark Structured Streaming paradigm • Experience working in a private equity-backed services company • Experience deploying machine learning models with MLFlow or equivalent • Experience developing CI/CD pipelines

Benefits

• 3+ years in data engineering roles in a production environment • Advanced proficiency in Python and SQL for data engineering • Up-to-date knowledge of and 1+ years of experience using Databricks for Lakehouse management • Deep understanding of data modeling, data architecture, and data integration best practices • Strong hands-on experience with Apache Spark • Familiarity with data governance, security, and privacy principles • Comfort using git or equivalent to manage the software development life cycle • Exceptional ability to learn and use new software development techniques and tools • Ability to manage multiple projects simultaneously • High energy, humble team player with "get it done" attitude, seeking collaboration with colleagues