AI for Research Efficiency, In Practice: Ask for the process, not the answer
Narration: an AI-generated voice (ElevenLabs).

[0:00] Three answers, one question
Three agent sessions got the same nine hundred and one survey comments, and the same request: tell me what researchers think.
They agreed on five broad themes. Only one put numbers on them, and only one made double-blind review a theme of its own.
With the method chosen and written down first, two separate runs agreed on ninety-nine percent of their coding decisions.
AI for Research Efficiency, In Practice. Ask for the process, not the answer.
In this film, an agent helps you choose the best way to do a research task, and saves it as a skill for next month.
It builds on episodes three, four, seven and eight.

[0:48] Why the answers differ
Ask twice and the answers differ. In one study, ChatGPT given the same instructions in separate conversations produced coding schemes with five to thirteen categories.
Anthropic and OpenAI both document this, and none of the three tools lists a setting to stop it.
People vary too. Twenty-nine teams of analysts, given one data set and one question, produced estimates from zero point eight nine to two point nine three: twenty found a significant effect, and nine did not.
A twenty twenty-six preprint finds that A I analysts' independent runs "frequently reverse whether the hypothesis is judged supported."

[1:31] 1. Ask for the process
So begin with a different question, in plan mode: Shift and Tab, or slash plan, in all three tools.
What is the best way to find out what these researchers think, in a way I could defend in a methods section and repeat next month? Don't analyze yet.
Then let it interview you. In Claude Code and Codex you ask for the interview in plain words; Antigravity has a command for it, slash grill me.
Here it asked eight questions. One answer changed the plan: next month's survey is on another topic, so the procedure, not a list of themes, is what must repeat.

[2:12] 2. Lay out the options
Next, three helpers each research one way to do the job, side by side.
One made the case for an agent coding against a fixed codebook, one for people coding by hand, and one for a topic model.
Keep the team small: Anthropic measured teams of agents using about fifteen times the tokens of a chat.

[2:36] 3. Test them on real comments
Then a reviewer tries each option on a sample of the real comments, and reports only the problems that would change the result.
Put that last part in your request: Claude Code's guide warns that a reviewer asked for gaps will usually report some, even when the work is sound.
On sixty real comments, it found that one for-or-against code does not fit, and that how requests for double-blind review are counted moves opposition from twenty-four to thirty-one percent.
It also found that the survey's authors had published an analysis of these same comments, in twenty seventeen.

[3:14] 4. Choose, and write it down
Now you choose. Here, the agent codes against a codebook: coding by hand would take about fifty hours, and the topic model's clusters barely repeated.
Before a single comment is coded, the agent writes the method down: a protocol with a decision log, and a codebook of nineteen reason codes.
Researchers call this preregistration. Plans still change, so each change goes in the log: in one journal, a review found that every preregistered study had departed from its plan.
This run changed two things before the freeze: one call per comment instead of three, to keep the cost down, and two new codes that a random pilot showed were missing.
Then it saves everything as a skill: a folder in dot agents slash skills for Codex and Antigravity, with a copy in dot claude slash skills for Claude Code.

[4:09] 5. Run it, then run it again
Check the first batch yourself, then let it run.
Run one coded eight hundred and twenty-one comments, each in a fresh call, in about fifteen minutes.
Then a fresh session, with nothing but the saved skill, codes every comment again.

[4:28] 6. Check it against something outside the agent
Here, the two runs agreed on ninety-nine percent of their coding decisions, and kappa, which discounts agreement by chance, averaged zero point nine four. Still, about one comment in five was coded differently somewhere.
That shows the method repeats. It does not show the codes are right: for that, compare them with a sample you code by hand.
So for a judgment like this one, the same result means an agreement rate you measure and report.
The paper is a check too: it counts nine hundred and one comments, as the file does, and the view it reports, double-blind review to cut bias, is the run's code for bias about authors, in fifty-five comments.
One trap: open the file with pandas, a popular Python library, on its default settings, and answers like None and N A become blanks. Nine hundred and one turns into eight hundred and eighty-three.

[5:26] Another field, and next month
By discipline, four fields have thirty or more substantive comments here, and their top reasons differ, often by just a few comments.
Read that with the survey's limits in view: one thousand two hundred and seventy-four of its three thousand and sixty-two respondents are earth and environmental scientists. The authors allow for some self-selection, and the survey dates from twenty sixteen.
And next month looks like this: the same skill, in Codex, on a newer survey of about two thousand four hundred researchers' reasons for their stance on a national open-data mandate. It drafted twenty-four reason codes, and stopped for the researcher's review, as the protocol says.

[6:09] The line not to cross
Use this on your own work, and on data you may share with these services; episode nine covers protected data.
And claim discovery modestly: helpers widen the menu, and you choose. In one study, ideas from language models were rated more novel than experts' ideas, and lost that edge once carried out.

[6:31] Try it
Pause here and try it. Before your next analysis, ask your agent: what is the best way to do this? Don't do it yet.

[6:43] Review and next
So: ask for the way, test the options, choose, write it down, and measure how far it repeats.
The skill and the practice folder are on the companion page.
This shelf of films was planned the same way: helpers researched which films would help researchers most, and a person makes the call.
Next on this shelf: make a slide deck that looks like yours.