Big Data Engineer

2 days ago

Apply Now

Receive Emails with Similar Jobs

Ness Digital Engineering

WebsiteLinkedInAll Job Openings

Systems Integration • Software Engineering Services • User Experience Engineering • User Experience Design • Software Product Development

1001 - 5000

Description

• Design & Build ingestion pipelines using ELT and schema on read in Databricks Delta lake • Design & Develop “Transformations” aspects using ELT framework to modernize the Ingestion pipelines and build data transformations at scale • Provide technical expertise in design and implementation of Ratings Data Ingestion pipelines with modern AWS cloud and other technologies • Build and maintain a data environment for speed, accuracy, consistency, and "up" time • Work closely with other data teams & Data Science team • Ensure data governance principles are adopted, data quality checks and data lineage are implemented • Be in tune with emerging trends Big data and cloud technologies • Ensure compliance through adopting enterprise standards and promoting best practices/guiding principles

Requirements

• A minimum of 8+ years of significant experience in application development • Previous experience in design and implementation of Ratings Data Ingestion pipelines with modern AWS cloud and other technologies such as S3, Hive, Databricks, Scala, Python, and Spark • Development, design, and architecture exposure • Proficient with software development lifecycle (SDLC) methodologies like Agile and test-driven development • Experience working with high-volume data and computationally intensive systems • Garbage collection friendly programming experience - tuning Java garbage collection & performance is a must • Proficiency in the development environment, including IDE, web & application server, GIT, Continuous Integration, unit-testing tools, and defect management tools • Domain knowledge in the Financial Industry and Capital Markets is a plus • Excellent communication skills are essential

Benefits

• Access to trainings and certifications • Bonuses • Aids • Socializing activities • Attractive compensation