Before you fix the question, check the label

I run a small research project: buyer questions in eight B2B categories, asked to an AI assistant with web search, then a second model reads each answer and lists every company it names. Count the names across answers and you get a rough map of who the assistants recommend in each category.

The first full pass produced two findings that looked like problems with my questions:

In two categories, my first readout said many answers were "advice without names": the assistant explained how to choose but didn't recommend anyone.
The name counts looked fragmented: 150 to 300 distinct names per 15 answers.

The obvious plan was to reword the questions in those two categories to ask for vendors directly. That would have been a mistake, and I almost made it.

Replay before you rewrite

The extractor's output was already stored, every answer and every list of names. So before touching a single question, I replayed the stored extractions through the counting code. No model calls, zero cost.

The "advice without names" answers each named 11 to 50 vendors.

The answers were fine. The label was wrong. The extractor had been asked to classify each answer's shape, and when an answer opened with "here's how to evaluate…" it called it advice, even when a long shortlist followed. The questions were doing their job.

Rewording them would have broken something real: I'm about to run a second pass to compare against the first. Change the questions and the two passes no longer measure the same thing.

Fixing the label, the right way

Two changes, one in the prompt and one in code:

The definition got sharper: "advice without names" only when the answer names no vendor at all.
Code backs it up. If the names kept from the answer's own text include a vendor, the label can't be "advice without names." When the model's label and the evidence disagree, the evidence wins.

The fragmentation was partly mine too

Reading the long tail of names showed two more counting bugs:

Variants counted separately. "Cambio" and "Cambio COSMIC" were two companies; so were "Archer" and "Archer BCM". The extractor now gives each name the form it should count under. As a backstop, code folds a name into a shorter name from the same pass that it starts with, word by word, but never into a root shorter than four letters, so "SAP Ariba" doesn't vanish into some three-letter acronym. Every fold is listed in the output so I can check it.
Non-vendors counted as vendors. Regulators ("check with the bar association"), public registers, and platforms mentioned only as something to connect to ("integrates with your ERP") were counted as recommendations. The extractor now classifies them separately: listed, but not counted.

The result

I re-counted the original pass with the new extractor. It's an additive run, so the old counts stay. The cost was about 65 cents for 120 answers.

Distinct names fell 15 to 30% per category, because variants are now one company.
"Advice without names" dropped from 3 to 4 answers per category to zero (one remains, in one category, and it's real).
The leaders became clearer, because their product lines no longer split their count.

One judgment call I flagged instead of hiding: in one category the extractor now treats a well-known ERP as "a platform the answer mentions" rather than a recommendation. That's sometimes right ("works with your ERP") and sometimes not. It's a known edge, written down, to check before anyone quotes that category.

What I took from it

Store what your model step returns. Re-reading stored output is free; re-asking isn't.
Replay before you rewrite. Most "the model is answering badly" problems I've seen were really "we're reading the answer badly."
Don't let a label overrule the evidence. If the model says "no names" and code finds names, believe the code.
Keep the change additive. New counts sit beside old ones, so both passes stay comparable.