Head of Evaluation and Oversight Research

Overview

Scale is the leading data and evaluation partner for frontier AI companies, playing an integral role in advancing the science of evaluating and characterizing large language models (LLMs). Our Research team shapes the next generation of evaluation science for frontier AI models and works at the leading edge of model assessment and oversight. Our work includes developing AI-assisted evaluation pipelines, advancing scalable oversight methods, and designing benchmarks for frontier capabilities, as well as building evaluation frameworks for agentic systems.

We collaborate broadly across industry and academia and regularly publish our findings.

Responsibilities

• Lead a team of research scientists and engineers on foundational work in evaluation and oversight.
• Drive research initiatives on frameworks and benchmarks for frontier AI models, spanning reasoning, coding, multi-modal, and agentic behaviors.
• Design and advance scalable oversight methods, leveraging model-assisted evaluation, rubric-guided judgments, and recursive oversight.
• Collaborate with leading research labs across industry and academia.
• Publish research at top-tier venues and contribute to open-source benchmarking initiatives.
• Remain deeply engaged with the research community, understanding trends and helping set them.

Ideal qualifications

Interview, culture, and expectations

Our research interviews are crafted to assess candidates' skills in practical ML prototyping and debugging, their grasp of research concepts, and their alignment with our organizational culture. We do not ask LeetCode-style questions.

Compensation, benefits, and location

Compensation packages at Scale for eligible roles include base salary, equity, and benefits. The base salary range for this position in the locations of San Francisco, New York, and Seattle is $260,000 — $350,000 USD. Your recruiter can share more about the specific salary range for your location during the hiring process, and confirm whether the role will be eligible for equity grant. Benefits include comprehensive health, dental and vision coverage, retirement benefits, a learning and development stipend, and generous PTO, with additional benefits such as a commuter stipend where applicable.

Please reference the job posting\'s subtitle for the location. We comply with the United States Department of Labor's Pay Transparency provision and Know Your Rights poster. We are an inclusive and equal opportunity workplace and do not discriminate based on race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability status, gender identity or Veteran status. We provide reasonable accommodations in the application process for applicants with disabilities. If you need assistance, contact accommodations@scale.com.

PLEASE NOTE : We collect, retain and use personal data for our professional business purposes, including notifying you of job opportunities that may be of interest and sharing with our affiliates. Any information collected will be treated in accordance with our privacy policy.

#J-18808-Ljbffr