Data Evaluator
About this opportunity
When a child types a question into a search box, the gap between what they typed and what they meant is usually wide. "Why is the sky blue", "how do volcanoes work", "is my hamster sick" - each has an intent behind it that the words only partly carry.
Search results evaluators close that gap. You will see a query and a set of results, work out what the person was actually trying to find, and rate how well each result meets that need. You will also judge the result itself: is the page trustworthy, is it written at a level the person could follow, and is it something we would be comfortable putting in front of a ten-year-old.
This is judgement work against a written rubric, not data entry. The rating matters, and the short written reason you give for it matters just as much - it is what makes the rating reviewable and what teaches the system.
No search-industry background is required. If you can read carefully, hold a rubric in your head and explain a decision in three sentences, you can do this work well.
What you'll do
- Interpret the intent behind a query, including ambiguous and misspelled ones
- Rate how well each result meets that intent against a written scale
- Assess page quality: who wrote it, whether it is accurate, whether it is trying to sell something
- Judge reading level and age-appropriateness for a child audience
- Flag results that are harmful, misleading, or dressed-up advertising
- Write a short, specific reason for every rating that another evaluator could follow
- Raise queries the rubric does not cleanly cover rather than forcing a rating
What we're looking for
- Excellent written English and careful reading
- Sound judgement on ambiguous questions with no single right answer
- Able to apply a detailed written rubric consistently across long sessions
- Comfortable assessing whether a web page is trustworthy
- Reliable internet connection and a computer
How the work is scheduled
- Fully asynchronous and pick up task batches when it suits you
- No minimum weekly hours and no fixed shifts
- Paid per hour of reviewed work, invoiced every two weeks
- Calibration tasks at the start of each project are paid
How we assess applications
A short paid calibration set, graded against the same rubric you would use on the job. We look at how consistent your ratings are and how clearly you justify them - not at how fast you work.
Good to know
- This is evaluation work on a children's product, not general content moderation - you will not be reviewing graphic or illegal material
- Queries occasionally touch sensitive subjects (illness, family difficulty). These are flagged in advance and you may decline any batch without explanation
- Locale matters: tell us which country's results you can judge confidently
- We are unable to support H1-B or STEM OPT contractors at this time.
Pay: $20.00 - $25.00 per hour
Expected hours: 15.0 – 20.0 per week
Benefits:
- Flextime
- Work from home
Work Location: Remote
