Talent.com
ML/LLM Operations Engineer
ML/LLM Operations EngineerEvolent • Nashville, TN, US
serp_jobs.error_messages.no_longer_accepting
ML / LLM Operations Engineer

ML / LLM Operations Engineer

Evolent • Nashville, TN, US
job_description.job_card.1_day_ago
serp_jobs.job_preview.job_type
  • serp_jobs.job_card.full_time
job_description.job_card.job_description

What You'll Be Doing :

Evolve your career by joining Evolent Health as an ML / LLM Operations Engineer. In this role you will monitor, evaluate, and ensure compliance for LLMs in healthcare settings, working across data science, engineering, and compliance teams to deliver reliable, regulatory-compliant AI services.

Collaboration Opportunities :

Partner with Data Science & Engineering, DevOps, Compliance, Quality Assurance, Clinical Support, and Product Management to keep AI systems operating reliably and meeting all healthcare sector requirements.

What You Will Be Doing :

Develop and maintain standardized evaluation frameworks to consistently measure LLM performance across relevant healthcare metrics.

Build monitoring systems using Logfire to track AI model performance, detect drift, and alert the team to anomalies.

Create testing infrastructure for prompt versions, model selection, and quality assurance processes.

Design and implement audit sampling processes for continuous quality monitoring and clinical review workflows.

Oversee regulatory compliance processes, including documentation for bias assessments, model cards, and audit trails required in healthcare.

Optimize LLM operations through intelligent model selection, prompt engineering, and cost management strategies.

Support the transition from successful POCs to production-ready services with appropriate testing and validation.

Partner with DevOps on infrastructure requirements while focusing on AI-specific monitoring and optimization.

Create and maintain documentation, runbooks, and operational procedures for all deployed AI systems.

Collaborate with Clinical Support Liaison to incorporate clinical feedback into system improvements.

Prepare regular reports on AI system quality, performance metrics, and compliance status.

Required Qualifications :

Bachelor's or master's degree in computer science, data science, or related field.

2+ years of experience with Python development and at least one production LLM implementation.

Strong proficiency in SQL for complex log analysis and metrics generation.

Demonstrated experience with LLM APIs and frameworks (experience with PydanticAI, LangChain, or similar).

Experience with monitoring tools and practices for AI systems, including performance metrics, drift detection, and alerting.

Understanding of LLM behavior, prompt engineering, and common failure modes in production.

Experience building evaluation or testing frameworks for AI / ML systems.

Strong communication skills for cross-functional collaboration.

Experience with healthcare AI applications and compliance requirements is preferred.

Familiarity with multiple LLM providers (OpenAI, Anthropic, Google, Azure) is preferred.

Knowledge of Pydantic ecosystem including PydanticAI and Logfire is preferred.

Understanding of LLM evaluation metrics and methodologies is preferred.

Experience building tools for non-technical users is preferred.

Basic knowledge of containerization (Docker) for local testing and development is preferred.

Experience with cloud environments (AWS, Azure) as a user is preferred.

Understanding of API rate limiting, quota management, and cost optimization strategies is preferred.

Knowledge of CI / CD concepts for ML model deployments is preferred.

Experience with regulatory compliance and audit processes is preferred.

Excellent documentation skills and attention to detail is preferred.

Evolent is an equal opportunity employer and considers all qualified applicants equally without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, or disability status.

The expected base salary / wage range for this position is $120,000 - $140,000. This position is also eligible for a bonus component that would be dependent on pre-defined performance factors. As part of our total compensation package, Evolent is proud to offer comprehensive benefits (including health insurance benefits) to qualifying employees. All compensation determinations are based on the skills and experience required for the position and commensurate with experience of selected individuals, which may vary above and below the stated amounts.

J-18808-Ljbffr

serp_jobs.job_alerts.create_a_job

Operation Engineer • Nashville, TN, US

Job_description.internal_linking.related_jobs
Lead Machine Learning Engineer

Lead Machine Learning Engineer

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Lead Machine Learning Engineer (REMOTE).Key Responsibilities : Design and lead ML architecture and model deployment strategies for batch and streaming use cases Ensure ...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Machine Learning Manager

Machine Learning Manager

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Machine Learning Manager (LLM).Key Responsibilities Manage a team of senior data scientists focused on ML-driven product development Collaborate with senior leadership...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Senior AI / ML Storage Engineer

Senior AI / ML Storage Engineer

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Senior AI and ML Storage Engineer.Key Responsibilities Design, develop, and operate distributed systems for managing data, compute, and networking for large-scale AI wo...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_variable_hours • serp_jobs.job_card.promoted • serp_jobs.job_card.new
Senior AI / ML Engineer

Senior AI / ML Engineer

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Senior AI / ML Engineer to design, develop, and deploy scalable AI / ML solutions in a mission-critical environment. Key Responsibilities Design and implement scalable machi...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Forward Deployed Engineer

Forward Deployed Engineer

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Forward Deployed Engineer, AI Agents.Key Responsibilities Work directly with customers to build and deploy AI agents using the developer toolkit Act as a technical adv...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Remote SIEM Engineer

Remote SIEM Engineer

VirtualVocations • Nashville, Tennessee, United States
serp_jobs.filters.remote
serp_jobs.job_card.full_time
A company is looking for a Cortex XSIAM Consultant to join a premier cyber security organization.Key Responsibilities Develop log ingestion strategy in collaboration with the technical lead Crea...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
MLOps Engineer

MLOps Engineer

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for an MLOps Engineer for the ML Recall team.Key Responsibilities Build, deploy, and maintain search services, including I / O-bound web services and data services Optimize sy...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
GTM Engineer

GTM Engineer

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Go-To-Market Engineer.Key Responsibilities Build and maintain custom integrations between GTM systems using APIs and automation tools Design and maintain ETL pipelines...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Machine Learning Engineer

Machine Learning Engineer

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for an Applied ML / LLM Engineer.Key Responsibilities Convert playbook checks into structured training datasets Fine-tune LLMs for clause classification, redline generation,...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Backend ML Engineer

Backend ML Engineer

VirtualVocations • Nashville, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Backend ML Engineer to build and maintain the backend infrastructure for an emotionally intelligent AI system. Key Responsibilities Own and enhance the backend infrastru...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Sales Operations Engineer

Sales Operations Engineer

VirtualVocations • Nashville, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Sales Operations Engineer to join their Sales Operations team.Key Responsibilities Build and maintain custom integrations between GTM systems using APIs and automation ...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Mid-Level Quality Engineer

Mid-Level Quality Engineer

VirtualVocations • Nashville, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Mid-Level Quality Engineer to enhance internal applications and ensure system performance.Key Responsibilities Troubleshoot and resolve issues through root cause analys...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Principal ML Engineer

Principal ML Engineer

VirtualVocations • Nashville, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Principal ML Engineer - Personalisation.Key Responsibilities Lead the architecture and evolution of scalable, high-performance personalization backend systems Drive cr...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_variable_hours • serp_jobs.job_card.promoted • serp_jobs.job_card.new
AI / ML Engineer

AI / ML Engineer

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for an AI / ML Engineer to act as a technical subject matter expert in Agentic AI.Key Responsibilities Design, build, and optimize AI Agents using advanced methods and assess t...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Operations Engineering Manager

Operations Engineering Manager

VirtualVocations • Nashville, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for an Operations Engineering Manager to lead continuous improvement initiatives and manage a team of Operations Engineers. Key Responsibilities Develop and implement a standa...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Site Reliability Engineer Team Lead

Site Reliability Engineer Team Lead

VirtualVocations • Nashville, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Site Reliability Engineer, Team Lead.Key Responsibilities Ensure 24x7 availability of production application systems Drive initiatives to improve operational efficienc...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Machine Learning Engineer

Senior Machine Learning Engineer

VirtualVocations • Nashville, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Senior ML Engineer - Personalisation.Key Responsibilities Develop scalable, high-performance personalization backend systems and AI-driven personalization pipelines Im...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
GTM Operations Specialist

GTM Operations Specialist

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a GTM Operations (Low-Code + AI Focus) teammate who is a business-savvy, technically fluent operator. Key Responsibilities Design scalable GTM systems connecting CRM, mark...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Software Engineer, AI / ML

Software Engineer, AI / ML

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a Software Engineer, Workspaces.Key Responsibilities Contribute to new product initiatives that assist ML engineers in analyzing and sharing information Collaborate with...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
GTM Operations Manager

GTM Operations Manager

VirtualVocations • Murfreesboro, Tennessee, United States
serp_jobs.job_card.full_time
A company is looking for a GTM Operations Manager to optimize systems and workflows for collaboration between Marketing, Sales, and Customer Success. Key Responsibilities Oversee and optimize the ...serp_jobs.internal_linking.show_more
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted