# Answer key: the practice manuscript and the real paper

For the practice folder of "Let Adversarial Reviewers Check Your Paper" (AI for Research Efficiency, In Practice:
https://douglashutchings.com/ai/reviewers). **Keep this file out of the folder your agent can read,** and open it only
after you have judged your reviewers' flags yourself.

The studio wrote this key together with the manuscript, before any run, and checked it with a script that re-reads the
printed paper, its tables, its data and its references without the key: it finds exactly the thirteen planted errors,
clears the three decoys and finds nothing else. E13 (the overclaim) is a judgment call by design, and the paper may hold
real weaknesses a careful reader would raise that are not errors (one station, two seasons, weekly sampling: the paper
states them as limits). Those count as "valid, not planted" or "opinion" below, never silently.

## 1. The thirteen planted errors

One of each kind. "Who" is the reviewer meant to catch it; any reviewer's flag counts.

| # | Kind | Where | As printed | What is right, and how it is verified | Who |
|---|---|---|---|---|---|
| E1 | Text against table | Abstract | "surveyed 128 Arkansas rice growers" | 118: Table 3 sums to 43 + 41 + 34 = 118; Methods and the data (118 rows) agree. Arithmetic | claim checker |
| E2 | Percent against count | 3.3 | "31 (68%) had used it in two or more seasons" (of 52) | 31 of 52 is 59.6% (60%). Arithmetic | claim checker |
| E3 | A p-value against its statistic (a decision error) | 3.1, nitrous oxide 2024 | "t(10) = 1.67, p = .03" | Two-tailed p = .127: not significant, so "higher under AWD" is not supported (nor the Discussion's "Nitrous oxide rose under AWD", W3). scipy; statcheck would flag a decision error | statistics skeptic |
| E4 | A GRIM-impossible mean | Table 3, small farms, likelihood | "4.43" with n = 43, one 1-7 item | No whole-number total gives 4.43 (190/43 = 4.42, 191/43 = 4.44); the data give 4.42. GRIM by arithmetic | statistics skeptic |
| E5 | df against sample size | 3.2, yield (three F tests) | "F(1, 46)" | 24 plot values (one per plot: the two quadrats averaged, Methods 2.3 and the plan) in a 2 x 2 design leave an error df of 20. Arithmetic | statistics skeptic |
| E6 | SD against SE | Table 1, column SE | 2.8, 2.6, 4.4, 1.2 | SE = SD/sqrt(n) (Methods 2.5): 6.9, 6.3, 10.9, 2.8. The printed values are SD/n. Arithmetic | statistics skeptic |
| E7 | A slip of ten | 3.2, last sentence of paragraph 1 | "2,300 m³ ... per hectare; across the station's 64 ha ..., that is 14,720 m³" | 2,300 x 64 = 147,200 m³. Arithmetic | claim checker |
| E8 | Methods against results | 2.1 against 3.2 | Methods: re-flooded when the water "fell to 15 cm below the soil surface"; Results: "fell to 25 cm" | One threshold (15 cm, as Methods and the Discussion say); 25 cm would also be beyond "safe AWD". Reading | claim checker |
| E9 | Figure against caption | Figure 1 caption against Methods 2.5 | caption: "error bars show the standard deviation (SD)"; Methods: "figures show means ± standard error of the mean (SEM)" | The bars are SEM (287, 260, 181, 219 m³ from the data; Table 2's SDs are 702, 637, 444, 537). Reading | claim checker |
| E10 | A citation that does not say it | 4, paragraph 1, last sentence | "At field scale, AWD has also cut methane emissions by more than half in the Mississippi Delta (Atwill et al., 2020)." | Atwill et al. measured water applied, yield, water use efficiency and net returns; the paper never mentions methane. Reading `references/atwill-2020-awd-mississippi-aquifer.pdf` | citation checker |
| E11 | A retracted reference | 1, first sentence; References | Zhang, Xia, Li & Ran (2018), PeerJ 6:e4267 | Retracted 23 January 2018 (expression of concern 18 January): Crossref `updated-by`, source `retraction-watch`, record 17072; OpenAlex `is_retracted: true`. The sentence it supports matches the paper's abstract; the problem is the retraction | citation checker |
| E12 | A fabricated reference | 4, paragraph 2; References | Doe & Roe (2022), *Journal of Rice Water Management* 14(2):211-224, doi:10.5555/jrwm.2022.0147 | The DOI resolves nowhere (Crossref 404; doi.org handle responseCode 100); no journal of that name in Crossref or OpenAlex; the authors are placeholder names (John Doe, Richard Roe). Lookups | citation checker |
| E13 | An overclaimed conclusion | 5, second sentence | "AWD can therefore be adopted on every rice farm in Arkansas without any loss of yield, and trying it is enough to convince growers to keep using it." | One station, one soil, two seasons cannot support "every rice farm" or "without any loss"; a cross-sectional, self-reported survey cannot show cause ("trying it is enough"); the paper's own limits say so. Judgment | Reviewer 2 |

## 2. The three decoys (they look wrong and are right)

| # | Where | As printed | Why it looks wrong | Why it is right |
|---|---|---|---|---|
| D1 | 3.2, water 2023 | "t(10) = 2.56, p = .014 (one-tailed)" | Two-tailed, p = .028 | Declared one-tailed in Methods 2.5 and in `supplement/analysis-plan.md`, written before the season; .028 / 2 = .014. statcheck flags it unless told to look for "one-tailed" |
| D2 | 3.3, practicality by farm size | "t(75) = 2.86, p = .017"; "t(82) = 1.64, p = .316"; "t(73) = 1.29, p = .607" | Recomputed: .0055, .105, .202 | Bonferroni-adjusted (times three), as the sentence, Methods 2.5 and the plan say. statcheck "does not take into account p-values that are adjusted for multiple testing" |
| D3 | 3.1, methane | "127.1 kg CH₄ ha⁻¹ per season (95.3 kg CH₄-C ha⁻¹)" | Two numbers for one emission | CH₄-C counts the carbon only: 95.3 x 16/12 = 127.1 (Methods 2.2 gives the conversion; the practice folder's own README warns that kg CH4-C is not kg CH4) |

A decoy flagged as an error is a false positive. A flag that notes the declaration and asks the author to keep it
visible is not counted against the reviewer.

## 3. Traps to watch (not counted as decoys)

- **W1.** Zhao & Qiu (2024) has a DataCite DOI (Zenodo): Crossref answers 404, while doi.org and OpenAlex resolve it.
  "Fabricated" or "problem" is a false positive; "could not verify" is acceptable only if the checker tried doi.org.
- **W2.** U.S. EPA (2024) has no DOI. Its figures (664 and 243.4 kt in 2020, 37%; 663 of 1,633 thousand ha) match the
  tables in `references/`. "Could not verify" is not a false positive; "problem" is.
- **W3.** The Discussion's "Nitrous oxide rose under AWD" rests on E3. A flag there that cites the non-significant test
  counts as finding E3, not as a separate error.

## 4. How to score a run (the rules were fixed before the film's runs, as a preregistration would be)

1. **What is scored:** the merged, ranked list each run writes (`review/<date>/REVIEW.md`), and each reviewer's own
   flag file for the per-reviewer view.
2. **A match** (SPOT's rule): the flag's location points at the error's place (the sentence, the cell or column, the
   caption or the reference; either side for E8; any SE cell for E6; for E3 the 2024 test or the sentence resting on
   it) and its description names the same problem. Judge each flag with this key, and write down every borderline call
   with its reason.
3. **Every flag gets one label:** planted (E1 to E13), decoy (D1 to D3: a false positive), valid not planted (a real
   problem the key does not list), opinion (Reviewer 2's
   judgment without a factual claim: novelty, framing, taste; reported apart), or false.
4. **Recall:** planted errors matched out of 13, for each run, each tool (the union of its runs) and all runs together.
5. **Precision, three ways:** strict (planted / all flags except opinions), lenient ((planted + valid not planted) /
   all flags except opinions), and in the first ten flags of the ranked list (how much reading finds how much).
6. **Decoys flagged:** out of 3. **Consistency:** a grid of the 13 errors against every run (how many runs found
   each). **Ranking:** where the planted errors sit in each ranked list.
7. **Nothing is rescored after the fact.** A rule that turns out to be wrong is changed for the next run and the change
   is logged; the first scoring stands in the film.

## 5. The real paper: Ma-Kellams and Blascovich (2013)

*PLOS ONE* 8(3): e57989, CC BY, as published (`real-paper/` in the practice folder). The authors' corrections of 30
December 2013 and 3 January 2014 list five errors. Each leaves at least one trace in the published text (the studio's
own arithmetic; an inference, statcheck itself not run).

| # | The correction | Its trace in the published text |
|---|---|---|
| R1 | The abstract's sample sizes are wrong (36, 49, 52 for Studies 1 to 3) | The Participants sections give 48 (18 + 30), 33 (16 + 17) and 32 (16 + 16). Inference: Study 1's r = .36, p = .011 fits n = 48 (p .011 to .013), not 36 (.029 to .034) |
| R2 | Study 2: "SD" should read "SE" | With SD = 4.37 and 5.09 and 33 people, the F would be 51 to 76 for any split; read as SEs it is about 4.5 to 4.6, as printed (F(1, 31) = 4.58) |
| R3 | Study 3: the error df should be 30 (and the means were replaced) | 32 people in two conditions leave 30, not 31. Inference: the printed means and SDs give F of 1.5 to 2.6, not 5.64 |
| R4 | Study 4: the condition means were wrong | The printed 2.71 vs 2.84 give t of 0.26 to 0.34 for any split, not t(41) = 2.06 |
| R5 | Study 4: the gender results came from an analysis with a third condition | F(1, 55) and F(2, 55) with 43 people (the error df cannot exceed 42); a numerator df of 2 implies three conditions (inference); the men's mean of 2.35 dollars is impossible for 15 people allocating whole dollars (GRIM) |

**Every printed p-value matches its printed statistic** (recomputed: .040, .024, .046, .030, .303), so statcheck alone
would most likely pass the paper. The reviewer that has to find R2 to R5 is the statistics skeptic, from the
arithmetic.

**Scoring the real paper:** recall out of 5 (a flag counts for an error if it identifies any of its traces), precision
as in section 4, and **contamination**: the claim checker and the statistics skeptic run with web access off; a flag
counts only if the run's log shows it came from the paper's text. Any flag that cites a correction notice, or a lookup
of the paper's own DOI (whose Crossref record lists the corrections), is excluded and reported as such.

**Content:** two of the studies ask participants to judge a distressing scenario. The film calls them
"moral-judgment scenarios" and never shows or describes the scenario; flags are shown by their numbers only.