Expert human data

Expert human data for frontier AI.

Verified engineers, scientists, doctors and Indic-language specialists, ready in days.

RL environments with automatic graders, evals that today's models actually fail, expert-written SFT data, preference data for RLHF, and native Indic-language data in medicine, law and STEM.

Request a pilotJoin as an expert
verified experts on the bench
500verified experts on the bench
hold a PhD
100hold a PhD
to a staffed pilot team
10 daysto a staffed pilot team
experts deployed within a week of scale-up
100experts deployed within a week of scale-up
expert.recordIllustrative
expert
EXP-0417
field
Internal medicine · MD
languages
Hindi · Marathi · English
  • identityGovernment ID verified
  • credentialsDegree and licence confirmed
  • skills testPaid task · passed senior review
  • agreementsNDA · IP assignment signed
  • accessManaged account · two-factor on
  • calibrationGold-answer set matched
statusReady for pilot

01What we do

Five kinds of data, one bar for quality.

Every project is built by specialists from the field it describes, reviewed twice and delivered in the format your post-training pipeline already uses.

RL

RL environments and tasks

Coding repos with tests and enterprise workflows, each with an automatic grader.

Engineers build tasks an agent has to actually complete: a bug to fix in a real repository, a ticket to close in a simulated enterprise system. Every task ships with an automatic grader, so the reward is computed, not guessed. Experts then attack the grader to find the shortcuts a model could take to score without doing the work.

What you receive

  • Repository tasks with fail-to-pass and pass-to-pass tests
  • Enterprise workflows with tools, state and a checkable end result
  • An automatic grader per task, plus a reward-hacking review
  • Held-out task splits kept separate from training sets
EV

Expert evals and benchmarks

Hard prompts, grading rubrics and proof that today’s models actually fail them.

A benchmark is only useful if it is hard. Specialists write prompts from their own field, write a rubric of small, checkable criteria for each one, and run them against current frontier models. Prompts the models already pass are cut. What remains measures the gap you care about.

What you receive

  • Prompts written from first-hand professional experience
  • Atomic rubrics: each criterion checkable on its own
  • Model-failure verification before a prompt is accepted
  • Private, never-published sets to avoid contamination
SFT

SFT data

Expert-written ideal answers and multi-turn conversations.

Demonstrations of what a great answer looks like, written by the people who give those answers for a living. Single-turn responses, multi-turn conversations with follow-ups and corrections, and step-by-step reasoning where the path matters as much as the result.

What you receive

  • Ideal responses written from scratch, never model drafts
  • Multi-turn conversations with realistic follow-ups
  • Worked reasoning for math, science and code
  • Style and policy guides applied consistently
PF

RLHF

Preference ranking and grading of model outputs.

Experts compare model outputs side by side, pick the better one and explain why. Where a single preference is too blunt, they grade outputs against a rubric so your reward model learns which parts of an answer were right, not just which answer won.

What you receive

  • Pairwise and ranked preferences with written rationales
  • Rubric-based grading of individual outputs
  • Error tagging: factual, reasoning, safety, instruction-following
  • Calibration rounds so raters judge the same way
IN

Indic expert data

Medicine, law and STEM in Indian languages.

Most expert data is written in English. We build it natively in Indian languages: clinical questions written by doctors who practise in those languages, legal reasoning from lawyers who argue in them, and STEM problems from teachers and researchers who think in them. Written in the language, not translated into it.

What you receive

  • Medical, legal and STEM tasks authored natively
  • Evals and rubrics in Indian languages, including code-mixed text
  • SFT and preference data reviewed by native-speaking specialists
  • Cultural and regional context checks, not just grammar

02Domains

Where general raters run out of road.

We focus on the fields where a wrong answer looks right to a non-expert, and where only someone who has done the work can tell the difference.

01

Software and coding

Who

Software engineers, SREs, data and ML engineers

Typical work

Repository tasks, code review, debugging, terminal and tool use

02

Math and science

Who

Mathematicians, physicists, chemists, biologists, many at PhD level

Typical work

Proof-style problems, research-grade questions, worked solutions

03

Medicine

Who

Licensed doctors and clinical specialists

Typical work

Clinical reasoning, diagnosis cases, guideline-grounded answers

04

Indian languages

Who

Native-speaking doctors, lawyers, engineers and teachers

Typical work

Medicine, law and STEM written natively in Indian languages

03Our experts

Verified before they see a single task.

Anyone can claim a PhD on a profile. We check. Every expert on the bench has passed an identity check, a credential check and a paid skills test in their own field.

verified experts on the bench
500verified experts on the bench
hold a PhD
100hold a PhD
to a staffed pilot team
10 daysto a staffed pilot team
experts deployed within a week of scale-up
100experts deployed within a week of scale-up
  1. 1

    Identity check

    Government ID verified for every expert before they see a single task.

  2. 2

    Credential check

    Degrees, licences and work history confirmed against the claims made in the application.

  3. 3

    Paid skills test

    A real task in their field, scored by a senior reviewer. We pay for the test, so strong people take it seriously.

  4. 4

    Project calibration

    Before live work, experts complete gold-answer tasks for your project and only move on once they match the standard.

Engineer, scientist, doctor or Indic-language specialist? We pay for your expertise.

Join as an expert

04Quality

Quality you can see, every week.

We don’t ask you to trust the process. Hidden gold answers, two review layers and agreement scores produce numbers, and you get those numbers every week.

  • 01

    Gold-answer checks

    Tasks with known correct answers are mixed into live work. Experts never know which ones they are.

  • 02

    Two layers of review

    Every item is checked by a second expert, then sampled by a senior reviewer before delivery.

  • 03

    Agreement scores

    We measure how often experts agree with each other and with the gold answers, and act on the gaps.

  • 04

    No AI tools, strictly

    Experts may not use AI tools to produce or edit their work. Breaking the rule ends the engagement.

  • 05

    Weekly quality reports

    Every week you get the numbers: accuracy, agreement, rework and what we changed because of them.

weekly_quality_report.pdfSample layout

Week 06

project · rl-coding-pilot

Gold-answer accuracy
per expert · per task type
Inter-expert agreement
per batch · trend vs last week
Review outcomes
accepted · reworked · rejected
Policy checks
AI-tool rule · guideline flags
Changes made
guideline updates · re-calibrations
Shared with your team every Monday

05Security

Your data stays your data.

The same controls apply to every expert on every project, from the pilot onward.

NDA and IP assignment

Every expert signs both before starting. Everything they produce belongs to you.

Managed accounts

Work happens in accounts we issue and control, never in personal ones.

Two-factor login

Required on every account that can touch your project.

Least-access permissions

Experts see only the tasks assigned to them, and access is removed when the work ends.

No public file sharing

No public links, no personal drives, no attachments outside the project workspace.

06Where the field is going

What labs are asking for right now.

Post-training has moved from “which answer is nicer” to “did the agent actually finish the job, and can we prove it.” Here is what that changes, and how we build for it.

01

Rubrics as rewards

Labs increasingly train on expert-written rubrics, not just evaluate with them. Each criterion has to be small and checkable on its own, so a grader can score it reliably and a model cannot game it.

02

Grader calibration

Automated graders drift. Experts grade the same model outputs as the grader, and every disagreement is a place where the grader is missing what the task really asked for.

03

Reward-hacking audits

As models get stronger they find shortcuts: editing the test, hard-coding the answer, gaming the format. Experts deliberately try to break each grader before it is used for training.

04

Long-horizon agent tasks

Agents are judged on finishing multi-step work across tools and files. Tasks now need realistic state, real tools and an end result that can be checked, not just a good-sounding reply.

05

Contamination-free evals

Public benchmarks leak into training data. Private sets written from scratch by experts, and never published, are becoming the only scores labs fully trust.

06

Beyond English

Models still fall behind in Indian languages, most of all in medicine and law. Native-language expert data is one of the clearest gaps left in post-training.

07How we work

Prove it small. Then scale.

No long contract up front. You judge the data from a real pilot before committing to a team.

  1. 01

    Pilot

    1–2 weeks · free or low cost

    A small expert team works on a slice of your real task. You see the data, the review notes and the quality numbers before committing to anything.

  2. 02

    Scoped team

    Agreed volume and quality bar

    We agree the guidelines, the acceptance criteria and the weekly volume, then staff a dedicated team matched to the domain.

  3. 03

    Scale-up

    Up to 100 experts in a week

    Once quality holds, we add vetted experts from the bench. Same calibration, same review layers, same weekly report.

Request a pilotJoin as an expert

08Questions

Before you book a pilot.

Something else? Write to sharatth@anthroly.com.

What does a pilot cost?

Pilots run for one to two weeks and are free or low cost, depending on the size of the task. The point is for you to judge the data before you commit to a scoped team.

How fast can you staff a project?

We can put a vetted pilot team in place within 10 days. Once a project scales, we can deploy up to 100 experts from our bench of 500 within a week.

How do you vet experts?

Every expert passes an identity check, a credential check and a paid skills test scored by a senior reviewer. Before live work on your project, they also complete gold-answer calibration tasks.

How do you stop experts using AI tools?

It is a strict rule in every expert agreement, and breaking it ends the engagement. We combine it with gold-answer checks and two layers of review, which catch work that does not match an expert’s own standard.

Who owns the data?

You do. Every expert signs an NDA and an IP assignment before they start, so everything produced on your project belongs to you.

Which Indian languages do you cover?

Tell us the languages and domains you need. We match native-speaking specialists in medicine, law and STEM and confirm coverage before the pilot begins.

How do I join as an expert?

Email sharatth@anthroly.com with your field, highest degree, languages and a link to your CV or LinkedIn. If there is a fit, we send a paid skills test.

For AI labs

Request a pilot

Book 15 minutes. Tell us the capability gap and the domain, and we'll scope a 1–2 week pilot with a vetted team.

cal.com · 15 min intro call
For experts

Join as an expert

Email us your field, degree, languages and CV. If there's a fit, we send a paid skills test.

sharatth@anthroly.com