WEBVTT 00:00:00.400 --> 00:00:04.331 Three agent sessions got the same nine hundred and one survey comments, 00:00:04.461 --> 00:00:07.890 and the same request: tell me what researchers think. 00:00:08.030 --> 00:00:10.215 They agreed on five broad themes. 00:00:10.586 --> 00:00:15.938 Only one put numbers on them, and only one made double-blind review a theme of its own. 00:00:15.988 --> 00:00:18.370 With the method chosen and written down first, 00:00:18.560 --> 00:00:22.803 two separate runs agreed on ninety-nine percent of their coding decisions. 00:00:26.800 --> 00:00:32.967 AI for Research Efficiency, In Practice. Ask for the process, not the answer. 00:00:36.300 --> 00:00:40.430 In this film, an agent helps you choose the best way to do a research task, 00:00:40.561 --> 00:00:42.926 and saves it as a skill for next month. 00:00:42.997 --> 00:00:46.733 It builds on episodes three, four, seven and eight. 00:00:48.400 --> 00:00:50.542 Ask twice and the answers differ. 00:00:51.113 --> 00:00:51.884 In one study, 00:00:52.054 --> 00:00:57.030 ChatGPT given the same instructions in separate conversations produced coding schemes 00:00:57.080 --> 00:00:59.483 with five to thirteen categories. 00:00:59.621 --> 00:01:02.372 Anthropic and OpenAI both document this, 00:01:02.503 --> 00:01:05.654 and none of the three tools lists a setting to stop it. 00:01:05.740 --> 00:01:06.922 People vary too. 00:01:07.393 --> 00:01:11.268 Twenty-nine teams of analysts, given one data set and one question, 00:01:11.379 --> 00:01:15.154 produced estimates from zero point eight nine to two point nine three: 00:01:15.645 --> 00:01:18.930 twenty found a significant effect, and nine did not. 00:01:18.980 --> 00:01:20.785 A twenty twenty-six preprint finds 00:01:20.835 --> 00:01:23.874 that A I analysts' independent runs "frequently 00:01:23.924 --> 00:01:27.172 reverse whether the hypothesis is judged supported." 00:01:31.600 --> 00:01:34.847 So begin with a different question, in plan mode: 00:01:35.178 --> 00:01:39.247 Shift and Tab, or slash plan, in all three tools. 00:01:39.417 --> 00:01:42.500 What is the best way to find out what these researchers think, 00:01:42.650 --> 00:01:48.275 in a way I could defend in a methods section and repeat next month? Don't analyze yet. 00:01:48.389 --> 00:01:50.170 Then let it interview you. 00:01:50.701 --> 00:01:54.574 In Claude Code and Codex you ask for the interview in plain words; 00:01:54.904 --> 00:01:58.407 Antigravity has a command for it, slash grill me. 00:01:58.577 --> 00:02:00.421 Here it asked eight questions. 00:02:00.893 --> 00:02:04.932 One answer changed the plan: next month's survey is on another topic, 00:02:05.083 --> 00:02:08.751 so the procedure, not a list of themes, is what must repeat. 00:02:12.400 --> 00:02:17.196 Next, three helpers each research one way to do the job, side by side. 00:02:17.258 --> 00:02:20.481 One made the case for an agent coding against a fixed codebook, 00:02:20.812 --> 00:02:24.395 one for people coding by hand, and one for a topic model. 00:02:24.532 --> 00:02:25.716 Keep the team small: 00:02:26.068 --> 00:02:30.924 Anthropic measured teams of agents using about fifteen times the tokens of a chat. 00:02:36.400 --> 00:02:40.248 Then a reviewer tries each option on a sample of the real comments, 00:02:40.359 --> 00:02:43.364 and reports only the problems that would change the result. 00:02:43.496 --> 00:02:45.461 Put that last part in your request: 00:02:45.772 --> 00:02:50.474 Claude Code's guide warns that a reviewer asked for gaps will usually report some, 00:02:50.625 --> 00:02:52.288 even when the work is sound. 00:02:52.338 --> 00:02:57.487 On sixty real comments, it found that one for-or-against code does not fit, 00:02:57.817 --> 00:03:01.935 and that how requests for double-blind review are counted moves opposition 00:03:01.985 --> 00:03:04.508 from twenty-four to thirty-one percent. 00:03:04.604 --> 00:03:09.132 It also found that the survey's authors had published an analysis of these same comments, 00:03:09.182 --> 00:03:10.484 in twenty seventeen. 00:03:14.800 --> 00:03:15.741 Now you choose. 00:03:16.392 --> 00:03:18.765 Here, the agent codes against a codebook: 00:03:19.236 --> 00:03:24.702 coding by hand would take about fifty hours, and the topic model's clusters barely repeated. 00:03:24.784 --> 00:03:28.532 Before a single comment is coded, the agent writes the method down: 00:03:28.863 --> 00:03:33.152 a protocol with a decision log, and a codebook of nineteen reason codes. 00:03:33.202 --> 00:03:35.546 Researchers call this preregistration. 00:03:36.037 --> 00:03:40.515 Plans still change, so each change goes in the log: in one journal, 00:03:40.605 --> 00:03:44.832 a review found that every preregistered study had departed from its plan. 00:03:44.906 --> 00:03:49.990 This run changed two things before the freeze: one call per comment instead of three, 00:03:50.040 --> 00:03:54.757 to keep the cost down, and two new codes that a random pilot showed were missing. 00:03:54.807 --> 00:03:56.909 Then it saves everything as a skill: 00:03:57.260 --> 00:04:01.514 a folder in dot agents slash skills for Codex and Antigravity, 00:04:01.785 --> 00:04:05.448 with a copy in dot claude slash skills for Claude Code. 00:04:10.000 --> 00:04:12.984 Check the first batch yourself, then let it run. 00:04:13.076 --> 00:04:15.942 Run one coded eight hundred and twenty-one comments, 00:04:16.133 --> 00:04:18.838 each in a fresh call, in about fifteen minutes. 00:04:18.988 --> 00:04:23.986 Then a fresh session, with nothing but the saved skill, codes every comment again. 00:04:29.200 --> 00:04:34.126 Here, the two runs agreed on ninety-nine percent of their coding decisions, and kappa, 00:04:34.257 --> 00:04:38.252 which discounts agreement by chance, averaged zero point nine four. 00:04:38.762 --> 00:04:42.727 Still, about one comment in five was coded differently somewhere. 00:04:42.817 --> 00:04:44.538 That shows the method repeats. 00:04:44.928 --> 00:04:50.292 It does not show the codes are right: for that, compare them with a sample you code by hand. 00:04:50.342 --> 00:04:55.723 So for a judgment like this one, the same result means an agreement rate you measure and report. 00:04:55.873 --> 00:05:00.957 The paper is a check too: it counts nine hundred and one comments, as the file does, 00:05:01.227 --> 00:05:04.540 and the view it reports, double-blind review to cut bias, 00:05:04.831 --> 00:05:08.994 is the run's code for bias about authors, in fifty-five comments. 00:05:09.084 --> 00:05:14.814 One trap: open the file with pandas, a popular Python library, on its default settings, 00:05:15.045 --> 00:05:18.641 and answers like None and N A become blanks. 00:05:19.072 --> 00:05:22.257 Nine hundred and one turns into eight hundred and eighty-three. 00:05:26.800 --> 00:05:30.892 By discipline, four fields have thirty or more substantive comments here, 00:05:31.103 --> 00:05:34.673 and their top reasons differ, often by just a few comments. 00:05:34.743 --> 00:05:36.824 Read that with the survey's limits in view: 00:05:37.334 --> 00:05:40.366 one thousand two hundred and seventy-four of its three thousand 00:05:40.416 --> 00:05:44.108 and sixty-two respondents are earth and environmental scientists. 00:05:44.198 --> 00:05:49.160 The authors allow for some self-selection, and the survey dates from twenty sixteen. 00:05:49.305 --> 00:05:53.226 And next month looks like this: the same skill, in Codex, 00:05:53.576 --> 00:05:57.428 on a newer survey of about two thousand four hundred researchers' reasons 00:05:57.478 --> 00:06:00.309 for their stance on a national open-data mandate. 00:06:00.679 --> 00:06:02.789 It drafted twenty-four reason codes, 00:06:02.879 --> 00:06:06.340 and stopped for the researcher's review, as the protocol says. 00:06:10.000 --> 00:06:11.384 Use this on your own work, 00:06:11.535 --> 00:06:16.508 and on data you may share with these services; episode nine covers protected data. 00:06:16.682 --> 00:06:21.410 And claim discovery modestly: helpers widen the menu, and you choose. 00:06:21.741 --> 00:06:26.580 In one study, ideas from language models were rated more novel than experts' ideas, 00:06:26.770 --> 00:06:28.974 and lost that edge once carried out. 00:06:31.600 --> 00:06:33.022 Pause here and try it. 00:06:33.313 --> 00:06:39.183 Before your next analysis, ask your agent: what is the best way to do this? Don't do it yet. 00:06:43.600 --> 00:06:50.978 So: ask for the way, test the options, choose, write it down, and measure how far it repeats. 00:06:51.191 --> 00:06:54.468 The skill and the practice folder are on the companion page. 00:06:54.538 --> 00:06:56.861 This shelf of films was planned the same way: 00:06:57.192 --> 00:07:02.258 helpers researched which films would help researchers most, and a person makes the call. 00:07:05.200 --> 00:07:08.936 Next on this shelf: make a slide deck that looks like yours.