1
00:00:00,400 --> 00:00:04,331
Three agent sessions got the same
nine hundred and one survey comments,

2
00:00:04,461 --> 00:00:07,890
and the same request:
tell me what researchers think.

3
00:00:08,030 --> 00:00:10,215
They agreed on five broad themes.

4
00:00:10,586 --> 00:00:15,938
Only one put numbers on them, and only one
made double-blind review a theme of its own.

5
00:00:15,988 --> 00:00:18,370
With the method chosen
and written down first,

6
00:00:18,560 --> 00:00:22,803
two separate runs agreed on ninety-nine
percent of their coding decisions.

7
00:00:26,800 --> 00:00:32,967
AI for Research Efficiency, In Practice.
Ask for the process, not the answer.

8
00:00:36,300 --> 00:00:40,430
In this film, an agent helps you choose
the best way to do a research task,

9
00:00:40,561 --> 00:00:42,926
and saves it as a skill for next month.

10
00:00:42,997 --> 00:00:46,733
It builds on episodes three,
four, seven and eight.

11
00:00:48,400 --> 00:00:50,542
Ask twice and the answers differ.

12
00:00:51,113 --> 00:00:51,884
In one study,

13
00:00:52,054 --> 00:00:57,030
ChatGPT given the same instructions in
separate conversations produced coding schemes

14
00:00:57,080 --> 00:00:59,483
with five to thirteen categories.

15
00:00:59,621 --> 00:01:02,372
Anthropic and OpenAI both document this,

16
00:01:02,503 --> 00:01:05,654
and none of the three tools
lists a setting to stop it.

17
00:01:05,740 --> 00:01:06,922
People vary too.

18
00:01:07,393 --> 00:01:11,268
Twenty-nine teams of analysts,
given one data set and one question,

19
00:01:11,379 --> 00:01:15,154
produced estimates from zero point
eight nine to two point nine three:

20
00:01:15,645 --> 00:01:18,930
twenty found a significant effect,
and nine did not.

21
00:01:18,980 --> 00:01:20,785
A twenty twenty-six preprint finds

22
00:01:20,835 --> 00:01:23,874
that A I analysts'
independent runs "frequently

23
00:01:23,924 --> 00:01:27,172
reverse whether the hypothesis
is judged supported."

24
00:01:31,600 --> 00:01:34,847
So begin with a different question,
in plan mode:

25
00:01:35,178 --> 00:01:39,247
Shift and Tab, or slash plan,
in all three tools.

26
00:01:39,417 --> 00:01:42,500
What is the best way to find out
what these researchers think,

27
00:01:42,650 --> 00:01:48,275
in a way I could defend in a methods section
and repeat next month? Don't analyze yet.

28
00:01:48,389 --> 00:01:50,170
Then let it interview you.

29
00:01:50,701 --> 00:01:54,574
In Claude Code and Codex you ask
for the interview in plain words;

30
00:01:54,904 --> 00:01:58,407
Antigravity has a command
for it, slash grill me.

31
00:01:58,577 --> 00:02:00,421
Here it asked eight questions.

32
00:02:00,893 --> 00:02:04,932
One answer changed the plan:
next month's survey is on another topic,

33
00:02:05,083 --> 00:02:08,751
so the procedure, not a list of themes,
is what must repeat.

34
00:02:12,400 --> 00:02:17,196
Next, three helpers each research one way
to do the job, side by side.

35
00:02:17,258 --> 00:02:20,481
One made the case for an agent
coding against a fixed codebook,

36
00:02:20,812 --> 00:02:24,395
one for people coding by hand,
and one for a topic model.

37
00:02:24,532 --> 00:02:25,716
Keep the team small:

38
00:02:26,068 --> 00:02:30,924
Anthropic measured teams of agents using
about fifteen times the tokens of a chat.

39
00:02:36,400 --> 00:02:40,248
Then a reviewer tries each option
on a sample of the real comments,

40
00:02:40,359 --> 00:02:43,364
and reports only the problems
that would change the result.

41
00:02:43,496 --> 00:02:45,461
Put that last part in your request:

42
00:02:45,772 --> 00:02:50,474
Claude Code's guide warns that a reviewer
asked for gaps will usually report some,

43
00:02:50,625 --> 00:02:52,288
even when the work is sound.

44
00:02:52,338 --> 00:02:57,487
On sixty real comments, it found
that one for-or-against code does not fit,

45
00:02:57,817 --> 00:03:01,935
and that how requests for double-blind
review are counted moves opposition

46
00:03:01,985 --> 00:03:04,508
from twenty-four to thirty-one percent.

47
00:03:04,604 --> 00:03:09,132
It also found that the survey's authors had
published an analysis of these same comments,

48
00:03:09,182 --> 00:03:10,484
in twenty seventeen.

49
00:03:14,800 --> 00:03:15,741
Now you choose.

50
00:03:16,392 --> 00:03:18,765
Here, the agent codes against a codebook:

51
00:03:19,236 --> 00:03:24,702
coding by hand would take about fifty hours,
and the topic model's clusters barely repeated.

52
00:03:24,784 --> 00:03:28,532
Before a single comment is coded,
the agent writes the method down:

53
00:03:28,863 --> 00:03:33,152
a protocol with a decision log,
and a codebook of nineteen reason codes.

54
00:03:33,202 --> 00:03:35,546
Researchers call this preregistration.

55
00:03:36,037 --> 00:03:40,515
Plans still change, so each change
goes in the log: in one journal,

56
00:03:40,605 --> 00:03:44,832
a review found that every preregistered
study had departed from its plan.

57
00:03:44,906 --> 00:03:49,990
This run changed two things before the freeze:
one call per comment instead of three,

58
00:03:50,040 --> 00:03:54,757
to keep the cost down, and two new codes
that a random pilot showed were missing.

59
00:03:54,807 --> 00:03:56,909
Then it saves everything as a skill:

60
00:03:57,260 --> 00:04:01,514
a folder in dot agents slash skills
for Codex and Antigravity,

61
00:04:01,785 --> 00:04:05,448
with a copy in dot claude
slash skills for Claude Code.

62
00:04:10,000 --> 00:04:12,984
Check the first batch yourself,
then let it run.

63
00:04:13,076 --> 00:04:15,942
Run one coded eight hundred
and twenty-one comments,

64
00:04:16,133 --> 00:04:18,838
each in a fresh call,
in about fifteen minutes.

65
00:04:18,988 --> 00:04:23,986
Then a fresh session, with nothing but the
saved skill, codes every comment again.

66
00:04:29,200 --> 00:04:34,126
Here, the two runs agreed on ninety-nine
percent of their coding decisions, and kappa,

67
00:04:34,257 --> 00:04:38,252
which discounts agreement by chance,
averaged zero point nine four.

68
00:04:38,762 --> 00:04:42,727
Still, about one comment in five
was coded differently somewhere.

69
00:04:42,817 --> 00:04:44,538
That shows the method repeats.

70
00:04:44,928 --> 00:04:50,292
It does not show the codes are right: for that,
compare them with a sample you code by hand.

71
00:04:50,342 --> 00:04:55,723
So for a judgment like this one, the same result
means an agreement rate you measure and report.

72
00:04:55,873 --> 00:05:00,957
The paper is a check too: it counts nine
hundred and one comments, as the file does,

73
00:05:01,227 --> 00:05:04,540
and the view it reports,
double-blind review to cut bias,

74
00:05:04,831 --> 00:05:08,994
is the run's code for bias about authors,
in fifty-five comments.

75
00:05:09,084 --> 00:05:14,814
One trap: open the file with pandas, a popular
Python library, on its default settings,

76
00:05:15,045 --> 00:05:18,641
and answers like None and N A become blanks.

77
00:05:19,072 --> 00:05:22,257
Nine hundred and one turns into
eight hundred and eighty-three.

78
00:05:26,800 --> 00:05:30,892
By discipline, four fields have thirty
or more substantive comments here,

79
00:05:31,103 --> 00:05:34,673
and their top reasons differ,
often by just a few comments.

80
00:05:34,743 --> 00:05:36,824
Read that with the survey's limits in view:

81
00:05:37,334 --> 00:05:40,366
one thousand two hundred
and seventy-four of its three thousand

82
00:05:40,416 --> 00:05:44,108
and sixty-two respondents are earth
and environmental scientists.

83
00:05:44,198 --> 00:05:49,160
The authors allow for some self-selection,
and the survey dates from twenty sixteen.

84
00:05:49,305 --> 00:05:53,226
And next month looks like this:
the same skill, in Codex,

85
00:05:53,576 --> 00:05:57,428
on a newer survey of about two thousand
four hundred researchers' reasons

86
00:05:57,478 --> 00:06:00,309
for their stance on a
national open-data mandate.

87
00:06:00,679 --> 00:06:02,789
It drafted twenty-four reason codes,

88
00:06:02,879 --> 00:06:06,340
and stopped for the researcher's review,
as the protocol says.

89
00:06:10,000 --> 00:06:11,384
Use this on your own work,

90
00:06:11,535 --> 00:06:16,508
and on data you may share with these services;
episode nine covers protected data.

91
00:06:16,682 --> 00:06:21,410
And claim discovery modestly:
helpers widen the menu, and you choose.

92
00:06:21,741 --> 00:06:26,580
In one study, ideas from language models
were rated more novel than experts' ideas,

93
00:06:26,770 --> 00:06:28,974
and lost that edge once carried out.

94
00:06:31,600 --> 00:06:33,022
Pause here and try it.

95
00:06:33,313 --> 00:06:39,183
Before your next analysis, ask your agent: what
is the best way to do this? Don't do it yet.

96
00:06:43,600 --> 00:06:50,978
So: ask for the way, test the options, choose,
write it down, and measure how far it repeats.

97
00:06:51,191 --> 00:06:54,468
The skill and the practice
folder are on the companion page.

98
00:06:54,538 --> 00:06:56,861
This shelf of films was
planned the same way:

99
00:06:57,192 --> 00:07:02,258
helpers researched which films would help
researchers most, and a person makes the call.

100
00:07:05,200 --> 00:07:08,936
Next on this shelf:
make a slide deck that looks like yours.