Open Source AI Engineer - Evals

February 20

Apply Now

Receive Emails with Similar Jobs

Arize AI

WebsiteLinkedInAll Job Openings

Arize AI is a company that provides a comprehensive AI observability and evaluation platform designed for AI engineers to build and manage AI applications, particularly those involving large language models (LLMs). The platform offers end-to-end AI tracing, monitoring, and evaluation tools to help users identify and resolve performance bottlenecks, ensure data quality, and improve the performance of AI systems. Arize also supports experimentation and visualization, enabling users to iterate rapidly on AI applications and maintain high-quality service delivery. The company caters to top AI companies, offering solutions for both online and offline LLM evaluation and observability. Arize's tools are particularly valuable for AI model monitoring, troubleshooting, and performance optimization, helping businesses mitigate risks and improve AI outcomes.

51 - 200 employees

Founded 2019

🤖 Artificial Intelligence

☁️ SaaS

🏢 Enterprise

📋 Description

• Build LLM Eval Frameworks: Design, architect, and open-source new libraries, pipelines, and APIs that make it simpler to evaluate LLM output quality, consistency, and reliability at scale. • Define Metrics and Benchmarks: Curate golden datasets and develop robust benchmarked metrics that guide data scientists and AI practitioners in optimizing their AI tasks. • Collaborate with the Community: Partner closely with the broader AI open source ecosystem, gather feedback, review pull requests, and steer the direction of the project to address real developer needs. • Prototype and Iterate Rapidly: Experiment with state-of-the-art LLM techniques, turning research into practical developer tooling. • Improve Observability and Debugging: Integrate with our existing platform to surface deeper insights on LLM behavior—help teams quickly diagnose and fix issues such as hallucinations or bias. • Educate and Evangelize: Write blog posts, white papers, tutorials, and documentation to help developers succeed with our open source tools and grow the LLM eval community.

🎯 Requirements

• Hands-on LLM Experience: Familiarity with popular LLM frameworks, prompt engineering techniques, and model fine-tuning. • Strong Programming Skills: Fluent in Python for AI workflows; bonus if you can navigate TypeScript as well. • Evaluation Knowledge: Understanding of core NLP evaluation methods and experience applying or extending them for LLM systems. • Open Source Track Record: Contributions to open source projects, personal GitHub repos with interesting AI demos, or a history of active engagement in developer communities. • ML Observability & Tools: Familiarity with debugging AI applications, exploring embeddings, or building data-heavy dashboards is a plus.

🏖️ Benefits

• medical • dental • vision • 401(k) plan • unlimited paid time off • generous parental leave plan • others for mental and wellness support