How we know what we show

The seven rules

1. Every number has a receipt. Each fact stores value, source, date checked, kind, confidence. If a number has no source, it doesn’t get published.
2. Advertised ≠ realistic ≠ reported. We keep three separate fields: what the platform advertises, what workers or founders report, and our derived “realistic hourly equivalent”. Never blend them silently.
3. Uncertainty is shown, not hidden. Confidence badges (A–D), “needs verification” banners and “as of” dates are visible on every card.
4. Ranking never depends on referral payouts. Match scores are computed without knowing whether we earn a referral. Publish this rule, and keep it true in the code: the matcher must not read the referral field.
5. Corrections are public. Every change lands in the changelog with date, source, method (automated or manual) and reviewer.
6. AI never invents facts. LLMs may extract facts only when they quote the supporting sentence from a fetched source, and code checks that the quote exists. LLMs may write explanations, but only from stored facts.
7. Fresh or flagged. Each entry has a review interval. Past it, the card shows “re-check due”, and the pipeline queues it.

Source tiers

Fact kinds (shown as small labels)

• official: the platform says it about itself (e.g., “Stripe primary, Wise fallback”)
• reported: press, an aggregator or research reports it (e.g., “median listed rate $75/h”)
• community: workers or founders self-report it (e.g., “$14–20/h effective”)
• claim: a marketing or podcast claim nobody checked (e.g., “70–95% gross margins”)
• estimate: our own derivation (e.g., ”≈ $688–6,450/month at 10 h/week”)

Facts vs. editorial guidance

Each entry has two layers, and the site should style them differently:

• Facts (the “Sourced facts” table, plus the earnings, time and cost numbers): claims with sources, kind, confidence and an as-of date. The pipeline watches these.
• Editorial guidance (one-liner, description, how it works, how to start, pros, cons, tips, red flags, trust signals, examples): practical advice written from the entry’s listed sources. It carries no confidence grade of its own.

Two rules keep the editorial layer honest:

1. No new statistics. Any money amount, rate or percentage in editorial text must already appear in that entry’s facts, earnings, costs or time fields. (Plain counts like “interview 10 operators” are advice, not claims.) If a number changes, the fact changes first, then the text. The build checks this automatically.
2. Reviewed like code. Editorial changes go through the same changelog. In the build, editorial text lives in separate files (x_enrich_*.py) and may only touch editorial fields.

Confidence grades (per fact and per entry)

• A: confirmed by a T1 source in this review cycle
• B: T2/T3 source, or a T1 source last checked 30–90 days ago
• C: T4 consensus, a single T3 source, or a podcast claim with a named source
• D: a single weak source (T5 or an uncorroborated T4), or seeded and not yet verified

Entry confidence is the grade of the facts that drive matching (pay, eligibility, status). In v1 (after the verification pass in doc 10): 5 A, 22 B, 31 C, 14 D across 72 entries.

Verification status

• verified_this_session: every fact was checked against its listed source on 2026-10-13
• partially_verified: some fields checked, others carried over
• seeded_needs_verification: a placeholder entry (7 in v1: User Interviews/Respondent, Contra, Toptal, italki, Cambly, translation marketplaces, SME Careers). Don’t publish its numbers until the pipeline verifies them.

Status states (badges)

active · active_turbulent (active, but with recent offboardings, bans, lawsuits or pay cuts) · waitlist · pilot · needs_verification · paused · closed (kept as a historical entry, e.g., Stellar AI, so users see how often platforms churn)

Freshness intervals (review_every_days)

What each card should show

Mercor · Confidence A · Active — turbulent · checked 2 days ago (re-check every 14 days)
Advertised $20–250/h · Realistic ≈ $16–150/h · Paid weekly (Stripe/Wise)
[Official] Payout schedule …          — Mercor talent docs, 2026-10-13
[Reported] Median listed rate $75/h  — aitrainer.work dataset, 2026-10-12
[Community] Offboarding wave (Jul 26) — Reddit via Breaking Even, 2026-08-09
Why we show it to you: … · What to watch out for: …

The changelog (public)

One row per change: date · entry · field · old → new · source · method (watcher / dataset / manual) · reviewer. Publish it as a page, an RSS feed and a weekly “What changed” post. It’s free content that proves the site is alive.

Corrections from users

There’s a “Report an issue” button on every fact. It goes into the review queue with the user’s evidence link. When a correction is accepted, the changelog credits the user (optional handle). This turns readers into your verification network.

How AI is used, and where it isn’t trusted

Lesson from auditing the ChatGPT reports

The three deep-research reports used as early input came with their citations stripped and several confident errors. Some examples:

• “Meridial/Invisible was founded by Sam Altman.” Invisible Technologies wasn’t founded by him.
• Toptal “backed by $225M funding.” Toptal is widely described as bootstrapped. This is unverified and likely wrong.
• Third Bridge “Nasdaq-listed.” This couldn’t be confirmed.
• OneForma described as “Pactera (former Lionbridge).” This couldn’t be confirmed.
• Outlier has “no official public referral program.” This is unverified.
• Mercor “average ~$109/hr.” That’s an advertised marketing figure, not average pay.

That is why rule 6 exists.

Known weaknesses of v1 (publish these too)

• Many business-model numbers are podcast or founder claims (confidence C/D). They’re labeled that way.
• Community sentiment comes partly from a third-party Reddit tally (Breaking Even), not your own collection yet.
• Pay for AI-training platforms varies by listing, country and week. Medians from listing datasets aren’t the same as realized earnings.
• Seven entries are placeholders awaiting verification.
• Starter Story revenue figures come from video titles and summaries (via an index), not from full transcripts.

Sources for this doc: chatgpt-dr-1 · chatgpt-dr-2 · chatgpt-dr-3 · aitrainer-dataset · breakeven-july · rosetta-starterstory · trustmrr-faq · revenuecat-2026 · mgmtsci-genai-freelance · upwork-fwi-2026 · gibson-omnibus · claude-pricing · prolific-pay