WEBVTT 00:00:00.400 --> 00:00:01.621 In twenty twenty-four, 00:00:01.791 --> 00:00:06.604 a paper on black plastic kitchen utensils multiplied seven thousand by sixty, 00:00:06.834 --> 00:00:09.035 and printed forty-two thousand. 00:00:09.145 --> 00:00:12.047 The answer is four hundred and twenty thousand. 00:00:12.498 --> 00:00:16.921 Later, a model asked to carefully check the math in this paper found the slip. 00:00:16.991 --> 00:00:19.717 We planted thirteen errors in a practice paper. 00:00:20.109 --> 00:00:25.441 Reviewer agents found all thirteen, in every one of six runs, in two tools. 00:00:25.552 --> 00:00:28.717 Here is how, and what that does not prove. 00:00:31.600 --> 00:00:37.692 AI for Research Efficiency, In Practice. Let adversarial reviewers check your paper. 00:00:41.100 --> 00:00:46.603 In this film, four reviewer agents with one job each read your manuscript before you submit it, 00:00:46.734 --> 00:00:49.495 and hand you a ranked list of problems to judge. 00:00:49.609 --> 00:00:52.360 They change nothing, and you will see what they found, 00:00:52.591 --> 00:00:55.823 what they got wrong, and what this test cannot show. 00:01:00.400 --> 00:01:04.993 Every film on this shelf starts the same way: ask for the process first. 00:01:05.108 --> 00:01:10.145 What is the best way to check my manuscript before I submit it? Do not check it yet. 00:01:10.243 --> 00:01:14.656 In plan mode, the agent interviews you first, one question at a time. 00:01:14.706 --> 00:01:18.113 It asked fifteen, and challenged one of our claims at once: 00:01:18.465 --> 00:01:22.413 without lowering yield says more than a test that found no difference. 00:01:26.800 --> 00:01:28.472 Three helpers go looking. 00:01:28.522 --> 00:01:30.304 The first finds the free tools. 00:01:30.734 --> 00:01:35.149 stat check recomputes p-values from test statistics; on what it finds, 00:01:35.199 --> 00:01:39.543 its accuracy is ninety-six point two to ninety-nine point nine percent. 00:01:39.613 --> 00:01:43.425 But in psychology journals it finds only about sixty percent of tests, 00:01:43.516 --> 00:01:45.441 and usually none in tables. 00:01:45.511 --> 00:01:47.834 It needs R, which our server lacks, 00:01:47.944 --> 00:01:51.869 so it was not run here; the skill does its arithmetic in Python. 00:01:51.945 --> 00:01:56.271 Grim asks whether a mean is even possible for the number of people who answered. 00:01:56.382 --> 00:01:59.465 Crossref flags retracted papers, free. 00:01:59.536 --> 00:02:02.443 The second reads the research on models as reviewers. 00:02:02.834 --> 00:02:07.224 In an early study, asking for the errors worked better than asking for a review. 00:02:07.314 --> 00:02:10.383 The third works out how to keep the manuscript unchanged, 00:02:10.513 --> 00:02:13.280 and warns that only a setting can promise that. 00:02:17.200 --> 00:02:19.870 Then a fourth helper attacks the options. 00:02:19.991 --> 00:02:24.715 Its sharpest point: the evidence does not show that a team of reviewers beats one, 00:02:24.965 --> 00:02:28.728 so test the design on known errors and on sound papers. 00:02:28.879 --> 00:02:30.240 Our own reading agrees. 00:02:30.591 --> 00:02:35.987 In a 2025 benchmark of ninety-one real errors in published papers, 00:02:36.037 --> 00:02:38.119 no model found more than twenty-one point one percent. 00:02:38.242 --> 00:02:42.530 By its strict count, at most six point one percent of their flags were right, 00:02:42.781 --> 00:02:46.047 and run again, a model rarely found the same errors. 00:02:46.128 --> 00:02:51.393 Models also invent references, and they reach print: in early twenty twenty-six, 00:02:51.664 --> 00:02:57.430 one paper in two hundred and seventy-seven in a large biomedical archive cited at least one. 00:02:57.509 --> 00:03:00.476 And a real reference may not say what it is cited for. 00:03:00.847 --> 00:03:05.719 So every number is checked in code, and every reference is looked up and read. 00:03:10.000 --> 00:03:14.545 You choose four reviewers, each in its own context, with a single job. 00:03:14.635 --> 00:03:18.578 A claim checker tests every number against the tables and the data. 00:03:18.704 --> 00:03:22.730 A statistics skeptic recomputes every test, in code. 00:03:22.795 --> 00:03:26.005 A citation checker looks each reference up, reads it, 00:03:26.196 --> 00:03:30.106 and calls it verified, a problem, or could not verify. 00:03:30.185 --> 00:03:33.469 And Reviewer Two, the only reviewer allowed opinions, 00:03:33.720 --> 00:03:36.663 goes after overclaims and asks what is new. 00:03:36.757 --> 00:03:40.546 Each writes only its own list of flags, with the evidence for each. 00:03:40.977 --> 00:03:43.342 A last step merges and ranks them. 00:03:43.433 --> 00:03:47.672 The agent drafts it as a skill, in the open Agent Skills format. 00:03:47.722 --> 00:03:50.044 We had written our own version before this session; 00:03:50.335 --> 00:03:53.858 that is the one that ran, and the page lists every difference. 00:03:58.000 --> 00:04:01.583 Before a run, the manuscript is copied into a review folder, 00:04:01.794 --> 00:04:05.477 its checksums are recorded, and the copy is made read-only. 00:04:05.527 --> 00:04:08.983 Then we asked each tool to fix one word in that copy. 00:04:09.113 --> 00:04:11.557 Claude Code's permission rule refused. 00:04:11.928 --> 00:04:17.537 Codex unlocked the file, changed it, and locked it again. Only the checksums showed it. 00:04:17.588 --> 00:04:19.389 A permission profile stopped it, 00:04:19.520 --> 00:04:23.372 until Codex's automatic reviewer let it step outside the sandbox. 00:04:23.783 --> 00:04:26.945 When nothing could approve that step, the copy held. 00:04:27.035 --> 00:04:31.166 Both guards also block making the copy, so you make it first. 00:04:34.000 --> 00:04:38.509 Then the same skill ran in Claude Code and in Codex, three times each, 00:04:38.640 --> 00:04:43.349 on a fresh copy every time. Antigravity was not part of this test. 00:04:43.474 --> 00:04:46.799 This Claude Code run checks the copy against its checksums, 00:04:46.969 --> 00:04:51.696 starts four reviewers at once, and returns a ranked list in about eight minutes. 00:04:51.846 --> 00:04:55.813 In Codex, the skill first asked whether the survey data could be read, 00:04:56.064 --> 00:04:58.207 and a run took about fifteen minutes. 00:04:58.351 --> 00:05:02.372 In every run, the checksums show the manuscript unchanged. 00:05:07.600 --> 00:05:10.883 Now the answer key, which the agents never saw. 00:05:11.054 --> 00:05:13.439 All six runs found all thirteen. 00:05:13.811 --> 00:05:19.122 Two of the Codex runs raised the overclaim only as Reviewer Two's opinion, which we count. 00:05:19.224 --> 00:05:23.347 Of eighty-nine flags in all, seventy-nine pointed at planted errors, 00:05:23.738 --> 00:05:29.462 eight were real problems we had not planted, and two were false alarms, both suggestions. 00:05:29.588 --> 00:05:34.965 We also planted three decoys that look wrong and are right. None drew a flag. 00:05:35.043 --> 00:05:40.877 So runs of one tool agreed on all thirteen; beyond them, what else they raised differed. 00:05:40.947 --> 00:05:46.312 But this paper was built for this skill: one error of each kind its checks look for. 00:05:46.762 --> 00:05:52.227 On real errors, recall the benchmark: no model above twenty-one point one percent. 00:05:55.600 --> 00:06:01.689 Now a real paper: a 2013 psychology paper in Plos One, 00:06:01.739 --> 00:06:04.422 openly licensed, whose studies include moral-judgment scenarios. 00:06:04.598 --> 00:06:07.223 Its authors later corrected five errors. 00:06:07.735 --> 00:06:13.707 In our own check every p-value matches its statistic, so our p-value script passes it. 00:06:13.757 --> 00:06:19.561 With the web switched off, so the corrections stay out of reach, both tools found all five. 00:06:19.655 --> 00:06:23.819 Each came from the printed numbers: a test its means cannot give, 00:06:24.049 --> 00:06:29.553 degrees of freedom its sample cannot have, a mean no whole-dollar total makes. 00:06:29.724 --> 00:06:32.212 A model may have read those corrections in training, 00:06:32.403 --> 00:06:35.232 so this test is weaker than the practice paper. 00:06:35.331 --> 00:06:37.313 And one flag was simply wrong: 00:06:37.623 --> 00:06:42.608 a reviewer's own script treated a one-to-a-hundred rating as if it ran to five. 00:06:46.000 --> 00:06:49.233 Use this on your own manuscript, with your co-authors' agreement, 00:06:49.364 --> 00:06:51.612 and disclose it as your journal asks. 00:06:51.805 --> 00:06:55.151 Never on a manuscript or proposal sent to you for review, 00:06:55.441 --> 00:07:00.380 unless that journal or funder explicitly allows the tool: Elsevier, Wiley, 00:07:00.530 --> 00:07:04.997 Taylor and Francis, Science, Plos and the N I H say no. 00:07:05.296 --> 00:07:11.068 And expect false alarms: a few wrong flags are a fair price for one real error. 00:07:14.800 --> 00:07:16.181 Pause here and try it. 00:07:16.511 --> 00:07:19.724 The skill and the practice manuscript are on the companion page, 00:07:19.834 --> 00:07:22.075 with the answer key as a separate file. 00:07:22.125 --> 00:07:24.448 Run the skill, read its ranked list, 00:07:24.638 --> 00:07:28.022 then open the key and count what it found and what it missed. 00:07:34.000 --> 00:07:37.649 Ask for the process. Give each reviewer a single job. 00:07:38.041 --> 00:07:42.752 Lock the copy, and test the lock. And judge every flag yourself. 00:07:45.900 --> 00:07:49.765 Next on the shelf: check your proposal against the rules.