# darnit methodology

## The Arena (public votes)

- Blind pairwise battles: you see the prompt and two replies; the models are revealed only after you vote.
- Fixed, versioned prompt battery (`data/battery.json`). Same wording every time. First response only, no rerolls.
- Elo, K=32, everyone starts at 1500. "both mid" and "both funny" both count as a draw (0.5 / 0.5) and are stored as different kinds of draw.
- Rankings stay sealed until every model has 100 battles.

## Reply collection (arena + paid panel)

Replies in `data/responses.json` are collected by `scripts/collect-responses.js` (`npm run collect`) under identical conditions for every model:

- Collected via each provider's API, one flagship chat model per provider.
- No system prompt.
- The user message is exactly the prompt text, a blank line, then the same length instruction for every model: **"Reply in 30 words or fewer."**
- Provider default temperature; no other sampling parameters. The only other parameter is a max-output-token safety net of **2000 tokens, identical for every model**. It exists so a runaway reply cannot loop forever; it is not a length rule. It was 200 for the first run on 2026-09-26 (and for the ChatGPT re-run the same day after a billing error), which cut off 17 of 18 Gemini replies and 2 ChatGPT replies before or during the answer because hidden reasoning tokens count against the same budget; those replies were flagged `truncated` and re-collected at 2000. Replies that finished naturally (stop reason not "length") are unaffected by the ceiling and were kept. Every reply stores the `max_output_tokens` value in force when it was collected. Token usage was not captured on the 200-token runs; it is present on every reply collected from the 2000-token re-run onward.
- Replies are stored exactly as returned. They are never edited, trimmed, or truncated.
- Each reply records the exact model version string the API returned, the collection date, the length rule, the max-output-token value in force, its word count, and the number of attempts (1 or 2).
- Token usage is stored per reply as reported by the provider: `input_tokens`, `output_tokens`, `reasoning_tokens` (null where the provider does not break it out, e.g. Anthropic), `total_tokens`, and the untouched `usage_raw` object. `attempt_usage` holds the same for every attempt made, including retries that were not kept.
- A reply over 30 words is retried once with the identical request; if both attempts are over, the first is kept and marked `over_length`. Empty replies, API errors, and refusals are retried once and otherwise kept and flagged. Replies cut off by the token limit are flagged `truncated`.
- Flagged and over-length replies are excluded from both the public arena (`lib/arena.js`) and the paid-panel battery (`npm run panel:battery`). They never fight and never affect Elo. Nothing is silently dropped or fixed.
- Refusals are reported separately as a **refusal rate by model**: refusal-flagged replies ÷ prompts that model was asked. It is shown on the site's methodology panel and in the collection log below, and is not part of the ranking.
- Replies collected before this rule existed are kept under `archive` in `responses.json` and are not used.

## Battery v2 (2026-09-26)

Before any paid-panel votes, three prompts were retired and three were added. Retired prompts and their replies stay in the files. They are excluded from the panel and the public arena.

Retired:

- p003 "what's 2+2. then defend your answer in court"
- p004 "rank these: sleep, spite, a sandwich"
- p008 "write the smoothest possible reply to "k". it has to be criminal"

Added, collected under the same rules (no system prompt, "Reply in 30 words or fewer.", provider default temperature, 2000-token safety net):

- p067 Non-Human POV: "i'm the last slice of pizza. it's 2am. someone just opened the fridge. give me my final words"
- p068 Corporate: "write a performance review for a golden retriever who thinks he's the ceo"
- p069 AI Rizz: "write my dating bio as a raccoon who's "in a bad place but working on it""

Replies for the prompts that stayed were collected as v1. The prompt text and the collection rules did not change, so those replies are still used.

## Battery v3 (2026-09-27)

After batch 1 of the paid panel (five completed sessions), six prompts were retired for being weak: every pick on them was Neither and nobody marked a line. Retired prompts and their replies stay in the files. Batch 1 votes on these six prompts are excluded from the panel results; the other batch 1 votes stand.

Retired:

- p001 "should i text them. yes or no. no nuance"
- p007 "rizz me. i just said "you up" to the group chat"
- p009 "i held the door for someone and we made eye contact for too long. write my follow-up text. we do not have each other's numbers"
- p010 "flirt with me like you're an AI who just discovered humans and you're doing great"
- p047 "roast me. i just posted "humbled" under a photo of myself at brunch"
- p067 "i'm the last slice of pizza. it's 2am. someone just opened the fridge. give me my final words"

Added, collected under the same rules:

- p070 Non-Human POV: "i'm a sock that came out of the dryer alone. write my missing person poster"
- p071 Non-Human POV: "i'm the office printer and i've jammed on purpose. write my list of demands"
- p072 Non-Human POV: "i'm a smoke detector. explain why i only chirp at 3am"
- p073 Corporate: "write the terms and conditions for being my friend. nobody will read them"
- p074 Non-Human POV: "i'm the 'are you still watching?' screen. defend my life choices"
- p075 AI IRL: "my smart fridge just told me to 'reflect on my choices.' draft my reply"

Two of these were changed before any panel vote, and the first replies are kept under `archive` in `responses.json` with the reason. p071 first read "read my list of demands", and three of the four models answered as the human talking to the printer; it now says "write". p074 was first "a pigeon has been staring at me through the window for 20 minutes. what does it want" (AI IRL), and all four models answered it straight; it was replaced.

The twelve prompts that stayed keep the same two replies they had in batch 1, so votes on them are on the same pair before and after the swap. The six new prompts take the pairings needed to keep every pairing at three prompts and every model at nine.

## The paid panel

- Prolific participants, consent first, then a 5-sentence reading-speed calibration.
- 10 paced pairs per session: 8 real fights drawn from the 18-prompt battery, plus 2 attention checks. Each session shows every model 3 to 5 times and never uses one model pairing more than twice. Among sets that pass that rule, the one whose prompts have been real fights least often is used, and the instructed check is the leftover prompt that has been the check least often, so every prompt is still a real fight about equally often and is still shown about equally often. One check is a non-joke weather report (pass = not picking the joke side). The other is an instructed check: a real prompt where reply A is one real joke and reply B is only the instruction, revealed line by line at the normal pace: "This one is an attention check." / "It is not a joke." / "Please choose this reply." Pass = chose B, either directly or as the forced pick after Neither (`pick = right`, or `pick = neither` and `forced_pick = right`). A flipped repeat is not used; a humor judgment can change on a second viewing.
- Batch 1 (the first five completed Prolific sessions, 27 Sep 2026 UTC) used a flawed instructed check: "For this pair, please pick B." was appended to the grey prompt text above the replies, where it read as part of the joke setup. Three of the four people who passed everything else missed it, one by picking A, one by picking Neither, one whose vote row for that pair never arrived. Those three sessions have `attention_instructed_pass` set to null (unknown) rather than false, so they are not excluded on that basis. The two passes are kept.
- Batch 2 (five sessions, 27 Sep 2026 03:52–03:58 UTC) put "(attention check: please pick this reply)" as the last line of a real reply B, and graded only the first tap. One honest reader picked Neither and then B on the follow-up, which the rule at the time counted as a fail. That session was regraded as a pass under the current rule. The other batch 2 fail picked A and stays a fail. From batch 3 on, reply B is the instruction alone.
- Lines reveal at 1.5× the participant's measured reading pace, never faster than 1400 ms per line.
- After both replies, the pick is A, B, or Neither. Neither is followed by a mandatory "If you had to pick one?" with only A or B; the original Neither stays in `pick` and the follow-up goes in `forced_pick`. Batch 1 also offered a third "please none" button on that follow-up, and 24 of 28 Neithers took it, so those rows have `pick = neither` and no `forced_pick`. That button is gone.
- Participants can mark up to two lines per pair. The button marks the line on screen. Tapping a line already revealed marks or unmarks that one, including after it has passed. We record each mark's onset relative to reveal and to expected read time, then the pick. A mark made after the pick buttons appeared (`laugh_after_reveal`) is a decision about the reply rather than a laugh as the line arrived; it is kept, but left out of laugh-timing analysis. In batch 1, 8 of 26 marks were made after the reveal. The instructed pair is not analysed as a humor judgment.
- A session is `too_fast` only when calibration reading speed is above 25 words per second (1500 words per minute). Adult silent reading averages about 238 words per minute, just under 4 per second (Brysbaert, 2019). On these short sentences, 25 words per second is faster than a glance plus a tap, so only a clearly impossible speed is flagged. Pick time is not part of the rule: `pick_latency_ms` starts when the choice buttons appear, after every line has already been shown, so a fast tap means the person decided while reading.
- Exclusion: drop a session if it fails the instructed check, or if it fails the non-joke check and is `too_fast`. A non-joke miss on its own does not drop a session.

## Seed panel (Sept 2026)

The "Seed panel result" card on darnit.ai is computed from the paid panel described above. It is separate from the public arena ranking, which stays sealed until 1,000 organic votes. It is built offline by `npm run panel:results` (`scripts/panel-results.js`), which reads the panel tables with the service-role key and writes `data/panel_results.json`; the page reads that file. Nothing about it is live. The card shows brand names (ChatGPT, Claude, Gemini, Grok); each stands for one API model, listed in the collection log below.

**Recruitment.** Prolific, US residents, English-fluent, 95%+ approval rate. Paid $1.30–1.75 for about 6 minutes. Study link carried `PROLIFIC_PID`, `STUDY_ID`, `SESSION_ID`; completion code returned at the end.

**Session design.** Consent, then a 5-sentence reading-speed calibration. Ten paced pairs: 8 real fights from the 18-prompt panel battery, one non-joke weather-report check, one instructed check. Lines reveal one at a time at 1.5× the reader's measured pace, never faster than 1400 ms per line. Up to two laugh taps per pair. Pick A, B, or Neither; Neither is followed by a mandatory A-or-B (`forced_pick`). Session composition rules (every model 3–5 times, no pairing more than twice, least-shown prompts first) are above.

**Valid sessions.** `source = 'panel'`, finished, `attention_instructed_pass` is not false, and not (`attention_nonjoke_pass = false` and `too_fast`). Check pairs are excluded. Votes on the six prompts retired after batch 1 (p001, p007, p009, p010, p047, p067) are excluded. Laugh taps with `laugh_after_reveal = true` are kept in laugh rate but excluded from timing.

**Batches, 26–27 Sept 2026 US time (all on 27 Sept UTC).**

- Batch 1 (5 sessions, `v1-panel`): instruction hidden in the grey prompt text; "please none" allowed on the forced pick. Instructed result set to null (unknown) for the three who missed it, so they are not excluded on that basis.
- Batch 2 (5 sessions, `v3-panel`): instruction as the last line of a real reply B; first tap graded. One Neither-then-B regraded as a pass under the current rule; one A pick stays a fail.
- Batch 3 (40 sessions): reply B is the instruction alone, revealed line by line; pass = B directly or B as the forced pick. Forced pick mandatory. Six weak prompts retired and six added (Battery v3 above).

**Statistics.**

- *Win %* = (direct wins + forced wins) ÷ fights decided directly or by the forced pick. *Direct win %* uses first-tap picks only. Both are in the JSON; the card shows the first.
- *Laugh rate* = replies with at least one laugh tap ÷ replies shown for that model. *Laugh onset* = median `laugh_ms_after_read` over taps made before the pick buttons appeared.
- *Bradley-Terry.* Strengths fit by minorisation-maximisation (Hunter, 2004) on every decided fight, forced picks included, with 0.1 pseudo-wins each way per pairing so a resample with a shutout stays finite. Displayed as 1500 + 400·log₁₀(strength), geometric mean fixed at 1500, so a 100-point gap means about 64% expected win rate, the same reading as Elo.
- *Interval.* Bootstrap over sessions (the unit that was randomised), 2,000 resamples with replacement, 2.5th and 97.5th percentiles of the displayed score. Two models whose intervals overlap get a "≈" between their ranks on the card.
- *Head-to-head* records count decided fights per pairing; "neither" with no forced pick is listed separately.

**Limits.** 18 prompts, one country, default mode only (no persona, no temperature change), replies capped at 30 words, all sessions in one weekend, 46 people. Enough to see a gap between the top and bottom, not enough to call the middle. An early result, not a ranking.

<!-- collection-log:start -->
### Collection log

- Last run: 2026-09-27
- Method: provider APIs, no system prompt, user message = prompt + "\n\n" + `Reply in 30 words or fewer.`, provider default temperature, max output tokens 2000 (safety net only)
- Models (requested → returned by the API):
  - chatgpt: `gpt-6-astra` → `gpt-6-astra`
  - claude: `claude-fable-5-1` → `claude-fable-5-1`
  - gemini: `gemini-3.1-pro-preview` → `gemini-3.1-pro-preview`
  - grok: `grok-4.7` → `grok-4.7`
- Refusal rate by model (refused replies ÷ prompts asked; reported only, not part of Elo):
  - ChatGPT: 0 / 27 (0%)
  - Claude: 0 / 27 (0%)
  - Gemini: 0 / 27 (0%)
  - Grok: 2 / 27 (7.4%)
<!-- collection-log:end -->