1
00:00:00,400 --> 00:00:01,621
In twenty twenty-four,

2
00:00:01,791 --> 00:00:06,604
a paper on black plastic kitchen utensils
multiplied seven thousand by sixty,

3
00:00:06,834 --> 00:00:09,035
and printed forty-two thousand.

4
00:00:09,145 --> 00:00:12,047
The answer is four hundred
and twenty thousand.

5
00:00:12,498 --> 00:00:16,921
Later, a model asked to carefully check
the math in this paper found the slip.

6
00:00:16,991 --> 00:00:19,717
We planted thirteen errors
in a practice paper.

7
00:00:20,109 --> 00:00:25,441
Reviewer agents found all thirteen,
in every one of six runs, in two tools.

8
00:00:25,552 --> 00:00:28,717
Here is how, and what that does not prove.

9
00:00:31,600 --> 00:00:37,692
AI for Research Efficiency, In Practice.
Let adversarial reviewers check your paper.

10
00:00:41,100 --> 00:00:46,603
In this film, four reviewer agents with one job
each read your manuscript before you submit it,

11
00:00:46,734 --> 00:00:49,495
and hand you a ranked
list of problems to judge.

12
00:00:49,609 --> 00:00:52,360
They change nothing,
and you will see what they found,

13
00:00:52,591 --> 00:00:55,823
what they got wrong,
and what this test cannot show.

14
00:01:00,400 --> 00:01:04,993
Every film on this shelf starts the
same way: ask for the process first.

15
00:01:05,108 --> 00:01:10,145
What is the best way to check my manuscript
before I submit it? Do not check it yet.

16
00:01:10,243 --> 00:01:14,656
In plan mode, the agent interviews
you first, one question at a time.

17
00:01:14,706 --> 00:01:18,113
It asked fifteen,
and challenged one of our claims at once:

18
00:01:18,465 --> 00:01:22,413
without lowering yield says more
than a test that found no difference.

19
00:01:26,800 --> 00:01:28,472
Three helpers go looking.

20
00:01:28,522 --> 00:01:30,304
The first finds the free tools.

21
00:01:30,734 --> 00:01:35,149
stat check recomputes p-values
from test statistics; on what it finds,

22
00:01:35,199 --> 00:01:39,543
its accuracy is ninety-six point two
to ninety-nine point nine percent.

23
00:01:39,613 --> 00:01:43,425
But in psychology journals it finds
only about sixty percent of tests,

24
00:01:43,516 --> 00:01:45,441
and usually none in tables.

25
00:01:45,511 --> 00:01:47,834
It needs R, which our server lacks,

26
00:01:47,944 --> 00:01:51,869
so it was not run here;
the skill does its arithmetic in Python.

27
00:01:51,945 --> 00:01:56,271
Grim asks whether a mean is even possible
for the number of people who answered.

28
00:01:56,382 --> 00:01:59,465
Crossref flags retracted papers, free.

29
00:01:59,536 --> 00:02:02,443
The second reads the research
on models as reviewers.

30
00:02:02,834 --> 00:02:07,224
In an early study, asking for the errors
worked better than asking for a review.

31
00:02:07,314 --> 00:02:10,383
The third works out how
to keep the manuscript unchanged,

32
00:02:10,513 --> 00:02:13,280
and warns that only a
setting can promise that.

33
00:02:17,200 --> 00:02:19,870
Then a fourth helper attacks the options.

34
00:02:19,991 --> 00:02:24,715
Its sharpest point: the evidence does not
show that a team of reviewers beats one,

35
00:02:24,965 --> 00:02:28,728
so test the design on known
errors and on sound papers.

36
00:02:28,879 --> 00:02:30,240
Our own reading agrees.

37
00:02:30,591 --> 00:02:35,987
In a 2025 benchmark of ninety-one
real errors in published papers,

38
00:02:36,037 --> 00:02:38,119
no model found more than
twenty-one point one percent.

39
00:02:38,242 --> 00:02:42,530
By its strict count, at most six point
one percent of their flags were right,

40
00:02:42,781 --> 00:02:46,047
and run again,
a model rarely found the same errors.

41
00:02:46,128 --> 00:02:51,393
Models also invent references, and they
reach print: in early twenty twenty-six,

42
00:02:51,664 --> 00:02:57,430
one paper in two hundred and seventy-seven in
a large biomedical archive cited at least one.

43
00:02:57,509 --> 00:03:00,476
And a real reference may not
say what it is cited for.

44
00:03:00,847 --> 00:03:05,719
So every number is checked in code,
and every reference is looked up and read.

45
00:03:10,000 --> 00:03:14,545
You choose four reviewers,
each in its own context, with a single job.

46
00:03:14,635 --> 00:03:18,578
A claim checker tests every number
against the tables and the data.

47
00:03:18,704 --> 00:03:22,730
A statistics skeptic
recomputes every test, in code.

48
00:03:22,795 --> 00:03:26,005
A citation checker looks
each reference up, reads it,

49
00:03:26,196 --> 00:03:30,106
and calls it verified,
a problem, or could not verify.

50
00:03:30,185 --> 00:03:33,469
And Reviewer Two,
the only reviewer allowed opinions,

51
00:03:33,720 --> 00:03:36,663
goes after overclaims and asks what is new.

52
00:03:36,757 --> 00:03:40,546
Each writes only its own list of flags,
with the evidence for each.

53
00:03:40,977 --> 00:03:43,342
A last step merges and ranks them.

54
00:03:43,433 --> 00:03:47,672
The agent drafts it as a skill,
in the open Agent Skills format.

55
00:03:47,722 --> 00:03:50,044
We had written our own version
before this session;

56
00:03:50,335 --> 00:03:53,858
that is the one that ran,
and the page lists every difference.

57
00:03:58,000 --> 00:04:01,583
Before a run, the manuscript
is copied into a review folder,

58
00:04:01,794 --> 00:04:05,477
its checksums are recorded,
and the copy is made read-only.

59
00:04:05,527 --> 00:04:08,983
Then we asked each tool
to fix one word in that copy.

60
00:04:09,113 --> 00:04:11,557
Claude Code's permission rule refused.

61
00:04:11,928 --> 00:04:17,537
Codex unlocked the file, changed it, and
locked it again. Only the checksums showed it.

62
00:04:17,588 --> 00:04:19,389
A permission profile stopped it,

63
00:04:19,520 --> 00:04:23,372
until Codex's automatic reviewer
let it step outside the sandbox.

64
00:04:23,783 --> 00:04:26,945
When nothing could approve
that step, the copy held.

65
00:04:27,035 --> 00:04:31,166
Both guards also block making the copy,
so you make it first.

66
00:04:34,000 --> 00:04:38,509
Then the same skill ran in Claude Code
and in Codex, three times each,

67
00:04:38,640 --> 00:04:43,349
on a fresh copy every time.
Antigravity was not part of this test.

68
00:04:43,474 --> 00:04:46,799
This Claude Code run checks
the copy against its checksums,

69
00:04:46,969 --> 00:04:51,696
starts four reviewers at once, and returns
a ranked list in about eight minutes.

70
00:04:51,846 --> 00:04:55,813
In Codex, the skill first asked
whether the survey data could be read,

71
00:04:56,064 --> 00:04:58,207
and a run took about fifteen minutes.

72
00:04:58,351 --> 00:05:02,372
In every run, the checksums
show the manuscript unchanged.

73
00:05:07,600 --> 00:05:10,883
Now the answer key,
which the agents never saw.

74
00:05:11,054 --> 00:05:13,439
All six runs found all thirteen.

75
00:05:13,811 --> 00:05:19,122
Two of the Codex runs raised the overclaim
only as Reviewer Two's opinion, which we count.

76
00:05:19,224 --> 00:05:23,347
Of eighty-nine flags in all,
seventy-nine pointed at planted errors,

77
00:05:23,738 --> 00:05:29,462
eight were real problems we had not planted,
and two were false alarms, both suggestions.

78
00:05:29,588 --> 00:05:34,965
We also planted three decoys that look
wrong and are right. None drew a flag.

79
00:05:35,043 --> 00:05:40,877
So runs of one tool agreed on all thirteen;
beyond them, what else they raised differed.

80
00:05:40,947 --> 00:05:46,312
But this paper was built for this skill:
one error of each kind its checks look for.

81
00:05:46,762 --> 00:05:52,227
On real errors, recall the benchmark:
no model above twenty-one point one percent.

82
00:05:55,600 --> 00:06:01,689
Now a real paper:
a 2013 psychology paper in Plos One,

83
00:06:01,739 --> 00:06:04,422
openly licensed, whose studies
include moral-judgment scenarios.

84
00:06:04,598 --> 00:06:07,223
Its authors later corrected five errors.

85
00:06:07,735 --> 00:06:13,707
In our own check every p-value matches its
statistic, so our p-value script passes it.

86
00:06:13,757 --> 00:06:19,561
With the web switched off, so the corrections
stay out of reach, both tools found all five.

87
00:06:19,655 --> 00:06:23,819
Each came from the printed numbers:
a test its means cannot give,

88
00:06:24,049 --> 00:06:29,553
degrees of freedom its sample cannot have,
a mean no whole-dollar total makes.

89
00:06:29,724 --> 00:06:32,212
A model may have read those
corrections in training,

90
00:06:32,403 --> 00:06:35,232
so this test is weaker
than the practice paper.

91
00:06:35,331 --> 00:06:37,313
And one flag was simply wrong:

92
00:06:37,623 --> 00:06:42,608
a reviewer's own script treated a
one-to-a-hundred rating as if it ran to five.

93
00:06:46,000 --> 00:06:49,233
Use this on your own manuscript,
with your co-authors' agreement,

94
00:06:49,364 --> 00:06:51,612
and disclose it as your journal asks.

95
00:06:51,805 --> 00:06:55,151
Never on a manuscript
or proposal sent to you for review,

96
00:06:55,441 --> 00:07:00,380
unless that journal or funder explicitly
allows the tool: Elsevier, Wiley,

97
00:07:00,530 --> 00:07:04,997
Taylor and Francis, Science,
Plos and the N I H say no.

98
00:07:05,296 --> 00:07:11,068
And expect false alarms: a few wrong
flags are a fair price for one real error.

99
00:07:14,800 --> 00:07:16,181
Pause here and try it.

100
00:07:16,511 --> 00:07:19,724
The skill and the practice
manuscript are on the companion page,

101
00:07:19,834 --> 00:07:22,075
with the answer key as a separate file.

102
00:07:22,125 --> 00:07:24,448
Run the skill, read its ranked list,

103
00:07:24,638 --> 00:07:28,022
then open the key and count what
it found and what it missed.

104
00:07:34,000 --> 00:07:37,649
Ask for the process.
Give each reviewer a single job.

105
00:07:38,041 --> 00:07:42,752
Lock the copy, and test the lock.
And judge every flag yourself.

106
00:07:45,900 --> 00:07:49,765
Next on the shelf:
check your proposal against the rules.