AI Agent Evaluation Analyst
MindriftUSjob_description.job_card.variable_days_ago
serp_jobs.job_preview.job_type
- serp_jobs.job_card.part_time
- serp_jobs.job_card.permanent
- serp_jobs.filters.remote
- serp_jobs.filters_job_card.quick_apply
job_description.job_card.job_descriptionReviewing evaluation tasks and scenarios for logic, completeness, and realism. Identifying inconsistencies, missing assumptions, or unclear decision points. Helping define clear expected behaviors (gold standards) for AI agents. Annotating cause-effect relationships, reasoning paths, and plausible alternatives. Thinking through complex systems and policies as a human would to ensure agents are tested properly. Working closely with QA, writers, or developers to suggest refinements or edge case coverage. Excellent analytical thinking : Can reason about complex systems, scenarios, and logical implications. Strong attention to detail : Can spot contradictions, ambiguities, and vague requirements. Familiarity with structured data formats : Can read, not necessarily write JSON / YAML. Ability to assess scenarios holistically : What's missing, what’s unrealistic, what might break? Good communication and clear writing (in English) to document your findings. Experience with policy evaluation, logic puzzles, case studies, or structured scenario design. Background in consulting, academia, olympiads (e.g. logic / math / informatics), or research. Exposure to LLMs, prompt engineering, or AI-generated content. Familiarity with QA or test-case thinking (edge cases, failure modes, “what could go wrong”). Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.). Get paid for your expertise, with rates that can go up to $55 / hour depending on your skills, experience, and project needs. Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments. Participate in an advanced AI project and gain valuable experience to enhance your portfolio. Influence how future AI models understand and communicate in your field of expertise.
This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.
At Mindrift , innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.
What we do
The Mindrift platform, launched and powered by Toloka , connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.
Who we're looking for :
We’re looking for curious and intellectually proactive contributors, the kind of person who double-checks assumptions and plays devil’s advocate.
Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?
This is a flexible, project-based opportunity well-suited for :
- Analysts, researchers, or consultants with strong critical thinking skills.
- Students (senior undergrads / grad students) looking for an intellectually interesting gig.
- People open to a part-time and non-permanent opportunity.
About the project :
We’re on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you’ll have to balance quality assurance, research, and logical problem-solving. This project opportunity is ideal for people who enjoy looking at systems holistically and thinking through scenarios, implications, and edge cases.
You do not need a coding background, but you must be curious, intellectually rigorous, and capable of evaluating the soundness and consistency of complex setups. If you’ve ever excelled in things like consulting, CHGK, Olympiads, case solving, or systems thinking — you might be a great fit.
What you’ll be doing :
How to get started :
Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone.
Requirements
We also value applicants who have :
Benefits
serp_jobs.job_alerts.create_a_job
Ai Analyst • US
Job_description.internal_linking.related_jobs
We are looking for Jira Developers with strong experience building custom plugins for Jira and Atlassian suite.You will be responsible for extending Jira functionality to meet specific business nee...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_1_day
Headquartered in Ottawa, with additional design center locations in Waterloo and San Jose, Fidus is a global leader and dynamic high-tech design firm.
Fidus specializes in leading-edge electronic pr...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.
We guarantee 15-25 hours per week with an hourly pay of bet...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30 Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.
We guarantee 15-25 hours per week with an hourly pay of bet...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30 
Experience working in Agile development shops Understanding of CI / CD methodology Experience with Test methodologies (4+ years) Familiarity with Azure (Preferred) or any cloud service (acceptable) E...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30
Want to make extra money on YOUR schedule? Join our exclusive list of research study participants and .Perfect for anyone seeking remote, part-time, or temporary work, these opportunities require ....serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_variable_days
This position is for individuals who want to develop a side income stream while still working full time.You will test different small-scale remote opportunities, learn what works, and grow what pro...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_variable_days
Position : SDET Engineer Location : Minneapolis, MN / Remote Duration : 1 Years Job Det...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_variable_days
Truity Credit Union is seeking a data-savvy and forward-thinking.Rather than full-scale software development, this position emphasizes.
You’ll collaborate with teams such as Software Engineeri...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_variable_days
Advanced knowledge and skills of IBM MQ technology.Accomplished technical MQ Expert in the depth of the knowledge, the level of skills, and the breadth of experience.
Able to serve as the technical ...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30 Cloud Security Services is seeking a security Customer Identity and Access Management (CIAM) Okta Engineer with background in global, complex, and diverse CIAM environments to design, de...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30 We're on the hunt for curious, motivated individuals to test and review online money-making opportunities featured on FinanceBuzz.
If you enjoy discovering new platforms and sharing your experience,...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30 
TherapyNotes is the go-to superhero for behavioral health Practice Management and EHR software! Our top-notch SaaS solution handles scheduling, billing, documenting, telehealth, and more so clinici...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30
The Test Lead is responsible for manual and automated testing of enterprise applications to ensure product quality and usability.
In this position, you will work closely with utility clients and imp...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30
The Voice of the Martyrs is a non-profit, inter-denominational Christian organization dedicated to assisting the persecuted church worldwide.
VOM was founded in 1967 by Pastor Richard Wurmbrand, who...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30 Want to make extra money on YOUR schedule? Join our exclusive list of research study participants and .Perfect for anyone seeking remote, part-time, or temporary work, these opportunities require ....serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_variable_days 
Here is a great job! Here is a great job!.Here is a great job!Here is a great job! Powered by JazzHR.serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30 Want to make extra money on YOUR schedule? Join our exclusive list of research study participants and start earning extra income today! Perfect for anyone seeking remote, part-time, or temporary wo...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_variable_days 
SAP solutions, dedicated to delivering innovative and high-quality services to businesses worldwide.Our mission is to empower organizations through effective and tailored SAP solutions, enabling th...serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_30 Want to make extra money on YOUR schedule? Join our exclusive list of research study participants and .Perfect for anyone seeking remote, part-time, or temporary work, these opportunities require ....serp_jobs.internal_linking.show_moreserp_jobs.last_updated.last_updated_variable_days
- serp_jobs.job_card.promoted
Jira Developer - Contractor
EYUSserp_jobs.job_card.full_time
Verification Designer
Fidus SystemsUSserp_jobs.filters.remote
serp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
- serp_jobs.job_card.promoted
Test Products from Home – $25-$45 / hr + Freebies
OCPANowata, Oklahoma, usserp_jobs.job_card.part_time +1
- serp_jobs.job_card.promoted
Remote Product Tester – $45 / hr + Free Products – Start Now!
OCPACherryvale, Kansas, usserp_jobs.filters.remote
serp_jobs.job_card.part_time +1
QA Tester
Cloud Security ServicesUSserp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
- serp_jobs.job_card.promoted
Wanted : Part-Time Work at Home Study Participants (Pay up to $3,000 / study)
MaxionUSserp_jobs.filters.remote
serp_jobs.job_card.full_time +2
- serp_jobs.job_card.promoted
Remote Side Hustle Developer
Finance BuzzBartlesville, Oklahoma, USserp_jobs.filters.remote
serp_jobs.job_card.full_time +1
SDET Engineer
Inherent TechnologiesUnited Statesserp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
- serp_jobs.job_card.promoted
Data & AI Engineer
Truity Credit UnionBartlesville, OK, USserp_jobs.job_card.full_time
MQ Expert
Two95 International Inc.USserp_jobs.filters.remote
serp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
Customer Identity and Access Management (CIAM) Okta Engineer
Cloud Security ServicesUSserp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
- serp_jobs.job_card.promoted
Work-from-Home Tester - Try Online Side Gigs and Give Feedback
Finance BuzzIndependence, Kansas, USserp_jobs.filters.remote
serp_jobs.job_card.temporary
Quality Assurance Engineer
TherapyNotes.comUSserp_jobs.filters.remote
serp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
Oracle CC&B Test Lead
Red Clay ConsultingUSserp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
- serp_jobs.job_card.promoted
PAS Representative - Data File Group
THE VOICE OF THE MARTYRSBartlesville, OK, USserp_jobs.job_card.full_time
- serp_jobs.job_card.promoted
Hiring : Part-Time Research Participants (Pay up to $790 / wk)
MaxionUSserp_jobs.job_card.full_time +2
Test job sensitive question
LinkedInUSserp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
- serp_jobs.job_card.promoted
Hiring : Part-Time Research Participants (Pay up to $3,000 / study)
MaxionUSserp_jobs.job_card.full_time +2
SAP ABAP Consultant-Remote (USA)
Sapsol Technologies IncUSserp_jobs.filters.remote
serp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
- serp_jobs.job_card.promoted
Wanted : Part-Time Work at Home Study Participants (Pay up to $790 / wk)
MaxionUSserp_jobs.filters.remote
serp_jobs.job_card.full_time +2