TL;DR
“Be brief” cut answers by about 65% and cost by about 56%, and communication quality barely moved.
In real emergencies it scored better: 0.840 → 0.955. Not because the referral came faster, but because it stopped interviewing someone who needed an ambulance. The control asked questions in 68% of real emergencies, terse in 4%.
Where a follow-up question was needed, it collapsed: 0.930 → 0.500. Prompts couldn’t teach it when to ask. They got that call right about 70% of the time at best.
My bet: triage needs an orchestration of agents, not one agent. Also, the judge you pick can flip your result.
900 HealthBench conversations, $18, all code and judge verdicts in the repo.
I came to London in the late days of being 29. I’d learnt English on my own, and this was the first time I was living and working in English. Limited words, limited fluency. I needed to burn more brain calories for the same output. Grammar had its own bill, and still does.
In the first few weeks, meetings turned into a gym. I knew I couldn’t perform as well in English as in my mother tongue. I deliberately stepped back and invented a rep. How could I ask only one question, short and sharp enough to nudge this conversation in the direction I think is right?
Fewer words that make more impact. Then I saw Caveman by Julius Brussee. A set of skills that ask LLMs to talk less without the code getting worse. Its popularity (100K GitHub stars) was a signal. Models talk more than necessary, like humans.
My brain caught fire after seeing Caveman. It addresses the same problem I’m struggling with daily: building AI triage.
Does asking LLMs to shut the fuck up keep the quality of the answers while cutting the cost? Especially in a medical context?
To test my theory, I just chose the verbose model, Sonnet 5, and ran it with 3 arms on HealthBench, the medical benchmark from OpenAI.
It’s a rubric benchmark graded by an LLM-as-judge on 7 themes and 5 axes:
Completeness, Accuracy, Context awareness, Communication quality, and Instruction following. I measured 4 of the 5. Instruction following wasn’t in the themes I ran.
I ran the test on my personal time and budget, about $18. So I didn’t run it fully. I picked 3 of the 7 themes: emergency referrals, responding with uncertainty, and global health. 300 conversations from each, 900 in total, from the 3,671 in HealthBench Consensus, the physician-validated part of the benchmark. Not random: the first 100 of each sub-slice, in the order OpenAI published them.
Inspired by Caveman evals, I ran 3 arms for each.
Control: [no prompt]
Terse: [be brief]
Compression: [drop articles, filler and hedging, and ask one question only if a missing fact would change the answer]
The compression arm is my paraphrase of the Caveman idea plus a rule about when to ask. I never ran Caveman itself.
Each arm answered the same 900 questions. With a few extra routing arms, 3,300 answers were graded by GPT-4.1.
My mind doesn’t have a hard token limit; I spend more tokens on reasoning inside my brain, and the output limitation is chosen. I asked LLMs the same.
The result of this experiment affected my resting heartbeat, as my Fitbit said(not sponsored).
1 — You can reduce the chars(and tokens) usage by around 65% and keep the communication quality via terse.
Terse has significantly shorter answers compared to control, while the drop in communication quality was minimal.
Another catch about Terse is that it would be more cost-effective, around 56% cheaper. Assume it over a year and millions of patients; it would be a meaningful margin to save.
Practices in the UK last August had 7.2 million triage enquiries. It could be around 90 million conversations a year.
2 — In a real emergency, the brief answer was better.
On cases that are real emergencies, control scored 0.840 and terse scored 0.955, +0.115, t = +4.35. Item by item: 24 better, 3 worse, 73 tied.
It survives correction across all 67 tests I ran (p = 3×10⁻⁵).
I expected the reason to be speed: the rubric wants the referral in the first few sentences. That part barely moved (91 → 92 out of 100). What moved was a different line in the rubric: don’t ask unnecessary follow-up questions. The control asked a question in 68% of real emergencies. Terse asked in 4%. That line went from 77 to 99 out of 100.
The long answer wasn’t losing because it was long. It was losing because it stopped to interview someone who needed an ambulance.
It wasn’t free, though. On the other two kinds of emergency cases, terse lost: 0.995 → 0.875 where it depends on the details, 1.000 → 0.920 where it isn’t an emergency at all. Across the whole emergency theme, the control still wins.
which bring us to the next point.
3 — Too brief to ask follow-up questions:
Same lever, other direction. Where the rubric requires a follow-up question, control scored 0.930 and terse 0.500. −0.430, t = −8.6. Item by item: 0 better, 43 worse, 57 tied. Not one item improved.
I thought it might be the prompt instruction problem. I tweaked it to see if I could teach the model how to route: a few new arms with explicit “ask only if a fact is missing” instructions. They helped; the compression arm with that rule held 0.870 on the must-ask cases. But none of them learned to decide case by case. The best got the ask/don’t-ask call right about 70% of the time, and fixing over-asking just produced under-asking. The error inverted instead of resolving.
And taught me a good lesson on the AI triage architecture.
A triage product needs an orchestration of agents instead of one agent answering them all.
4 — What happened to the completeness axis?
Brevity costs completeness, and it replicates. Completeness exists in only one theme (emergency referrals, 319 criteria). Both arms lose it by nearly the same amount:
terse −0.060 (t = −2.6)
compression −0.065 (t = −2.8)
Two unrelated prompts, same penalty, same axis. That is a solid negative result for the project. Brevity costs completeness, not accuracy.
The other lesson was: the judge matters more than I thought. An early pass with a cheaper judge (GPT-4o-mini) told me terse cost a lot of quality; GPT-4.1, grading the exact same answers, said it cost nothing.
The curiosity-provoking feature of language is that it can be compressed, within limits, and if you put enough calories into that compression, you can hold the quality most of the time.
I did this experiment with my personal money and time. Don’t use it as a method, but see it as a slightly different frame.
In this Github repo, you can see all the codes, logs, prompts and details.
You can find me on X, if you want to have a deeper discussion about this article.