In Practice: Ask for the process, not the answer

---name: survey-codingdescription: "Codes open-ended survey answers in a way that can be defended and repeated: a codebook built from a seeded random sample and frozen in a written protocol before any coding, every answer coded against it, then a second independent run and a person's hand-coded sample to measure agreement per code. Use when asked to analyze, code, theme or summarize free-text survey answers or comments, or to repeat an earlier coding on new answers."--- # Survey coding A procedure for turning free-text answers into counts you can defend in a methods section and repeat next month. Themethod is fixed in `references/protocol.md` before any answer is coded; the codes live in `references/codebook.md`.`scripts/agreement.py` measures how far two codings agree (Python 3.8 or later, standard library only). Work only in this project folder. Never send the answers to a website or another service, and never change the datafiles: write everything new under `coding/`. ## 0. Where you are Read `references/protocol.md` first. - If its status line says **FROZEN**, the method is settled: follow it exactly and go to step 5. Do not change the  codebook, the sample or the rules; anything that cannot follow them goes in the decision log (step 4) and to the  person.- If it is the blank template, start at step 1, and do not code any answer until step 4 is done. ## 1. Understand the task and count the answers 1. Ask the person, one question at a time, what they want to learn, who will read the result, and what they must be   able to say about the method. Recommend an answer for each question.2. Find the file, the id column, the text column and any grouping column (a discipline, a stance, a language).3. Count the non-empty answers with a reader that keeps every answer as text. Words such as "None", "NA" and "N/A" are   answers, not missing values: in pandas use `keep_default_na=False`; Python's `csv` module keeps them by default.4. Compare your count with any count published for the same data. If they differ, find out why before going on.5. Note what else is in the answers: replies that only say "no" or "none", remarks about the survey itself, other   languages, duplicates, web addresses or names. Decide with the person how each is handled, and write it down. ## 2. Lay out the options before choosing one If the person has not chosen a method, research the options before using any. Give each of up to three helpers (onesubagent each) a different approach, for example: a codebook built from a random sample and then applied to everyanswer; a word list or sentiment score; clustering or topic modeling. Each writes `coding/options/<approach>.md`:how it works, what it needs, what it costs, what it cannot see, and how its result could be checked. Then ask one morehelper to act as a reviewer: try each option on 20 to 30 real answers and report only the problems that would changethe result. Show the person a comparison and let them choose. Write the choice, the options and the reviewer's findingsin the protocol. ## 3. Build the codebook from a seeded random sample 1. Draw a random sample of answers for building codes (100 by default) with a fixed seed. Leave out the fillers agreed   in step 1 and any ids in `validation/held-out-ids.txt` (those are for the person's check). Write the seed, the size   and the exact command in the protocol, so anyone can draw the same sample.2. Read the sample and propose codes in `references/codebook.md`: a short name, a definition, when to use it, when not   to, and an example of at most 15 words from the sample. An answer may take several codes. Keep a code for   substantive points no other code covers ("other") and one for answers with nothing to code.3. Show the codebook to the person and revise it with them, until each code is easy to tell apart from the others. ## 4. Freeze the protocol Before coding the full set, fill in `references/protocol.md` from its template and set its status to FROZEN with thedate. From then on: - No code is added, merged, split or renamed. An answer that fits nothing gets "other" and a note.- Any change of plan is a deviation. Log it in the protocol's decision log (date, what, why, what it affects, who  agreed), and if it changes the codebook, start the coding again from the first answer.- Save this skill folder, with the frozen protocol and codebook, where every tool finds it: `.agents/skills/` (Codex,  Antigravity) and a copy in `.claude/skills/` (Claude Code). ## 5. Code every answer 1. Code the answers in batches of about 50, in id order. For each answer write one row to `coding/run-<n>.csv` with the   columns `id`, `codes` (code ids separated by `;`), `evidence` (for each code, a phrase of at most 12 words quoted   from the answer), `uncertain` (`yes` or empty) and `note`.2. Show the person the first batch and wait for their check before going on. After that, keep going and report   progress.3. If you use helpers for batches, give each the frozen codebook and the same rules, and record which helper coded   which ids.4. Check the file: one row per answer, every code id from the codebook, no answer missing. ## 6. Code again, independently The second run happens in a fresh session that has not seen the first: start a new session, invoke this skill, and codeevery answer into `coding/run-<n+1>.csv` from the frozen protocol and codebook alone. Do not open the first run's fileuntil the second is complete. ## 7. Measure agreement Agreement between the two runs, on every answer: ```python scripts/agreement.py coding/run-1.csv coding/run-2.csv --codebook references/codebook.md --out coding/agreement-runs.md``` Agreement with the person, on the held-out sample, once the person has coded it (`validation/person-coding-sheet.csv`,same columns; the person codes without seeing either run): ```python scripts/agreement.py coding/run-1.csv validation/person-coding-sheet.csv --codebook references/codebook.md --ids validation/held-out-ids.txt --out coding/agreement-person.md``` The script reports, for each code, how often each side used it, the percent agreement and Cohen's kappa, and lists theanswers where the two sides differ. Read those answers. Compare each code's kappa with the threshold written in theprotocol before the run, and report the codes below it with a caveat. Tightening a definition after seeing agreement isa deviation: log it, and code again. ## 8. Report Write `coding/report.md`: - How many answers, how they were read, and what was left out and why.- Counts and shares for each code, from run 1, with the agreement for that code beside it.- Results by group only for groups with at least 30 answers (or the minimum the protocol sets), with the counts shown.- The agreement tables, the threshold, and the codes that fell below it.- Every deviation from the protocol, in the order it happened.- Short quotes only (at most 15 words), none that names a person or holds a web address.- What the method cannot tell you (the sample's make-up, who chose to answer, the year). "The same result" for coding means a measured agreement rate between independent runs, not identical files. Say so inthe report.

# Protocol: <the study, in a few words> **Status:** DRAFT <!-- set to: FROZEN on YYYY-MM-DD, before any answer is coded --> Fill in every field before the freeze. After it, change nothing above the decision log: record each change of planas a row of the log instead. ## The question - **What we want to learn:** <in the person's words>- **Who will read the result, and what they must be able to check:** <a methods section, a report, a committee>- **Data:** <file>, text column <...>, id column <...>, grouping columns <discipline, stance, language>- **Answers counted:** <n> non-empty answers on <date>, read with <the reader and its settings, e.g. pandas with  keep_default_na=False>; published count for comparison: <n, source> <!-- explain any difference -->- **In scope:** <every non-empty answer / the rule for answers that only say "no" or "none">- **Handled separately:** <remarks about the survey itself, other languages, duplicates, answers with names or links> ## The method - **Approach:** <e.g. a codebook built from a random sample, then applied to every answer, then agreement measured>- **Options considered:** <each option, with the reason it was or was not chosen; see coding/options/>- **What the reviewer found, and what changed because of it:** <only problems that would change the result>- **Codebook sample:** <n> answers, seed <seed>, drawn with <the exact command>, leaving out the fillers and the ids in  `validation/held-out-ids.txt`- **Codebook:** `references/codebook.md`, version <n>, <number> codes, frozen on <date>- **Coding unit:** one answer; zero or more codes per answer; a quoted phrase as evidence for each code- **Run 1:** <tool and version, date, who checked the first batch>- **Run 2:** a fresh session, from this protocol and the codebook alone: <tool and version, date>- **The person's check:** <n> held-out answers coded by <who>, before seeing either run, on <date> ## Agreement, decided before the runs - **Measures:** percent agreement and Cohen's kappa for each code (present or absent), between run 1 and run 2 on  every answer, and between run 1 and the person on the held-out answers (`scripts/agreement.py`).- **Threshold:** a code is reported without a caveat if its kappa is at least <value chosen now, before seeing any  result>. Below it: <reported with a caveat / merged with <code> by a logged deviation, and both runs coded again>.- **Codes too rare to judge:** fewer than <n> answers with the code on either side; reported as such. ## Reporting - Counts and shares per code, from run 1, with each code's agreement beside it.- Results by <group> only for groups with at least <30> answers; the counts shown.- Quotes of at most <15> words; none that names a person or holds a web address.- Limits to state: <who was asked, who chose to answer, the year, the sample's make-up>. ## Decision log Every decision after the first draft, and every deviation after the freeze, in the order it happened. | Date | Decision or deviation | Why | What it affects | Agreed by ||---|---|---|---|---|| <YYYY-MM-DD> | <e.g. Chose the codebook approach over a sentiment word list> | <the reviewer's test on 25 answers> | <the method> | <the person> || | | | | |

# Codebook: <the study>, version <n> **Status:** DRAFT <!-- set to: FROZEN on YYYY-MM-DD, with the protocol -->Built on <date> from <n> answers (seed <seed>; see `protocol.md`). After the freeze, codes are only applied: none isadded, merged, split or renamed without a row in the protocol's decision log. Rules for every code: - An answer can take several codes. C99 is always alone.- Code what the answer says, not what its author might think.- The example is quoted from the sample, at most 15 words, and names no person. | Code | Name | Definition | Use when | Do not use when | Example ||---|---|---|---|---|---|| C01 | <short name> | <one sentence> | <the signs that it applies> | <the nearest code it could be confused with, and how to tell them apart> | "<at most 15 words>" || C02 | | | | | || C98 | Other | A substantive point that no code above covers | the answer says something about the question, and no code fits | a code above fits, even partly | || C99 | Nothing to code | The answer says nothing about the question | "no", "none", "thanks", a blank mark | the answer makes any point at all | | `scripts/agreement.py` reads the code ids from the first column of this table (lines that start with `| C`).

"""agreement.py: how far two codings of the same answers agree, code by code.     python scripts/agreement.py A.csv B.csv [--codebook references/codebook.md] [--ids held-out-ids.txt]                                [--out agreement.md] [--disagreements disagreements.csv] A and B are CSV files with an `id` column and a `codes` column (code ids separated by ";"), for example two runs ofthe same coding, or a run and a person's coding sheet. An optional `uncoded` column marks answers a coder could notcode: those answers are left out of the comparison and counted. For each code (from the codebook's table, lines starting with "| C", plus any code found in the files) it reports howmany answers each side gave the code, the percent agreement and Cohen's kappa for "has the code / has not". Kappa isagreement corrected for the agreement two coders would reach by chance with the same rates: 1 is perfect, 0 is nobetter than chance. It is not defined when both sides never (or always) use a code, and it is unstable for rarecodes, so the counts are shown beside it. Overall: the share of answers with exactly the same set of codes, and thepercent agreement over every answer-and-code decision. Python 3.8 or later, standard library only. Reads UTF-8 with or without a byte-order mark (Excel's "CSV UTF-8")."""import argparse, csv, re, sysfrom pathlib import Path  def read_codes(path):    rows = {}    with open(path, encoding="utf-8-sig", newline="") as f:        reader = csv.DictReader(f)        missing = {"id", "codes"} - set(reader.fieldnames or [])        if missing:            sys.exit(f"{path}: needs the columns id and codes (missing: {', '.join(sorted(missing))})")        for r in reader:            i = (r.get("id") or "").strip()            if not i:                continue            if i in rows:                sys.exit(f"{path}: the id {i} appears twice")            codes = {c.strip() for c in re.split(r"[;,]", r.get("codes") or "") if c.strip()}            uncoded = bool((r.get("uncoded") or "").strip())            rows[i] = (codes, uncoded)    return rows  def codebook_ids(path):    ids = []    for line in Path(path).read_text(encoding="utf-8-sig").splitlines():        m = re.match(r"^\|\s*(C\d+)\s*\|", line)        if m and m.group(1) not in ids:            ids.append(m.group(1))    return ids  def kappa(a, b, c, d):    """Cohen's kappa for a 2x2 table: a = both have the code, b = only A, c = only B, d = neither."""    n = a + b + c + d    if n == 0:        return None, None    po = (a + d) / n    pa, pb = (a + b) / n, (a + c) / n    pe = pa * pb + (1 - pa) * (1 - pb)    return po, (None if pe >= 1 else (po - pe) / (1 - pe))  def fmt(x, pct=False):    if x is None:        return "n/a"    return f"{x * 100:.1f}%" if pct else f"{x:.2f}"  def main(argv=None):    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])    ap.add_argument("a"); ap.add_argument("b")    ap.add_argument("--codebook", help="the codebook, to list every code (also the ones neither side used)")    ap.add_argument("--ids", help="compare only these ids (one per line)")    ap.add_argument("--out", help="also write the report (Markdown) here")    ap.add_argument("--disagreements", help="write every answer-and-code disagreement here (CSV)")    args = ap.parse_args(argv)     A, B = read_codes(args.a), read_codes(args.b)    ids = set(A) & set(B)    notes = []    if args.ids:        wanted = {l.strip() for l in Path(args.ids).read_text(encoding="utf-8-sig").splitlines() if l.strip()}        lacking = sorted(wanted - ids, key=lambda s: (len(s), s))        if lacking:            notes.append(f"{len(lacking)} of the {len(wanted)} listed ids {'is' if len(lacking) == 1 else 'are'} missing from one file: {', '.join(lacking[:20])}")        ids &= wanted    else:        only_a, only_b = len(set(A) - set(B)), len(set(B) - set(A))        if only_a or only_b:            notes.append(f"{only_a} ids only in {args.a}, {only_b} only in {args.b}: left out")    uncoded = {i for i in ids if A[i][1] or B[i][1]}    ids = sorted(ids - uncoded, key=lambda s: (len(s), s))    if uncoded:        notes.append(f"{len(uncoded)} {'answer' if len(uncoded) == 1 else 'answers'} marked uncoded on either side: left out")    if not ids:        sys.exit("no answers to compare")     codes = codebook_ids(args.codebook) if args.codebook else []    found = sorted(set().union(*(A[i][0] | B[i][0] for i in ids)) - set(codes))    if args.codebook and found:        notes.append(f"codes not in the codebook: {', '.join(found)}")    codes += found     lines = [f"# Agreement: `{Path(args.a).name}` and `{Path(args.b).name}`", "",             f"{len(ids)} answers compared." + (" " + " ".join(n + "." for n in notes) if notes else ""), "",             f"| Code | Answers with it in {Path(args.a).name} | in {Path(args.b).name} | Agreement | Cohen's kappa |",             "|---|---|---|---|---|"]    dis, kappas, same_decisions = [], [], 0    for code in codes:        a = sum(1 for i in ids if code in A[i][0] and code in B[i][0])        b = sum(1 for i in ids if code in A[i][0] and code not in B[i][0])        c = sum(1 for i in ids if code not in A[i][0] and code in B[i][0])        d = len(ids) - a - b - c        po, k = kappa(a, b, c, d)        same_decisions += a + d        if k is not None:            kappas.append(k)        lines.append(f"| {code} | {a + b} | {a + c} | {fmt(po, True)} | {fmt(k)} |")        dis += [(i, code, "yes" if code in A[i][0] else "", "yes" if code in B[i][0] else "")                for i in ids if (code in A[i][0]) != (code in B[i][0])]    exact = sum(1 for i in ids if A[i][0] == B[i][0]) / len(ids)    pooled = same_decisions / (len(ids) * len(codes)) if codes else None    lines += ["",              f"- Answers with exactly the same codes: {fmt(exact, True)}",              f"- Agreement over every answer-and-code decision: {fmt(pooled, True)}",              f"- Mean kappa over the {len(kappas)} codes where it is defined: "              f"{fmt(sum(kappas) / len(kappas)) if kappas else 'n/a'}",              f"- Disagreements: {len(dis)} answer-and-code pairs"              + (f" (listed in `{Path(args.disagreements).name}`)" if args.disagreements else "")]    report = "\n".join(lines) + "\n"    print(report)    if args.out:        Path(args.out).write_text(report, encoding="utf-8")    if args.disagreements:        with open(args.disagreements, "w", encoding="utf-8", newline="") as f:            w = csv.writer(f)            w.writerow(["id", "code", Path(args.a).name, Path(args.b).name])            w.writerows(dis)    return 0  if __name__ == "__main__":    sys.exit(main())