The hand check was the spec

I run an agent that watches a company's market every morning. It reads the public web around that market and works out what changed since yesterday. Only then does it ask a model which changes matter, and why.

The split is on purpose. Code fetches, measures and compares. The model only judges, and it gets the measured facts as givens it cannot change. A quiet morning costs nothing, because when nothing is new, no model is asked.

The first version went live over a weekend. It had a test for every path I could think of. It ran clean.

The test I did not write

In its third week I did the same market check by hand, the slow way: page by page, source by source, reading everything the agent should have read.

The hand check found three items that mattered that week. The agent's log had none of them. It also had four pages it called new that were months old.

So I laid the two side by side, item by item, and asked one question of each gap: what in the code made this happen?

A hand check that beats your system is a free spec. Diff against it.

Every gap was a quiet assumption

None of the gaps was a crash or an exception. Each one was a reasonable assumption that held on the happy path and failed without a sound.

What the code assumed	What was true
A failed read and an empty page are the same	One is "I could not see it", the other is "I saw nothing"
A site is one thing	A site is many files and feeds, and each one fails on its own
The day I first saw a page is the day it appeared	Most pages say when they were made
Many new pages from one site are many findings	A rebuild is one event
The headline says who a story is about	The names were in the body
A skip list means the same thing everywhere	A list written for one lane hid signal in another
Its most important input would always arrive	It came through one door, and went stale in silence

An absence is not an observation

The first row caused the old pages. Deep in the fetch layer, a request that failed and a page that came back empty looked the same to everything above them. The likely story: one part of a rival's site did not load on the first day. When it loaded later, everything in it looked brand new.

The fix is small and boring, which is the point. The fetch says whether it read anything at all:

pythondef fetch(url: str) -> tuple[str, bool]:
    try:
        r = session.get(url, timeout=25)
    except RequestException:
        return "", False
    if r.status_code >= 400:
        return "", False
    return r.text, True

Everything above it can now tell "nothing there" from "could not look". A part that did not load sets no memory. Its contents wait for the day it does.

If a system can't tell "I could not see it" from "I saw nothing", it will invent news.

Keep state at the grain where things fail

I had kept "what I have seen" per site. But a site fails in pieces, so the memory has to live in pieces too. The same goes for dates. "New to me" and "new to the world" are different facts. When a page carries its own date, the page wins over the moment I happened to look.

Memory belongs to the smallest thing you read. Dates come from evidence, not from when you looked.

Cheap filters decide what the expensive judge sees

One story that mattered was fetched and then dropped before the model ever saw it. A cheap relevance check looked at the headline and summary, and the names it wanted were only in the body.

That is the trap with a cheap filter in front of an expensive judge. Every item the filter drops is an item the judge never gets to keep. The filter has to be at least as right as the judge.

A filter in front of a model is part of the model's judgement. Test it like one.

The rebuild

I did not patch symptoms. I wrote the hand check out as a spec, step by step, and for each step decided one thing: can code decide this on its own, or does a person have to judge it?

Most of it is code: comparing, dating, counting, checking whether a name is on a page and in what order. Framing, cause and what to do next stay with a person. That line matters more than any single fix. Code that pretends to judge is as dangerous as a model that pretends to measure.

Then I replayed the week as tests. Each failure became a fixture shaped like the real one: the part that failed on day one and loaded on day two, the old page that looked new, the burst, the story with the names below the fold. The tests call the agent the way production does, defaults included. I learned that one the hard way months ago, when a bug lived for weeks in the one path no test called.

Before the new code ran, I migrated the stored memory once, adding rows and never rewriting them. What the agent has already gathered is the record, and the record is not edited.

Where it stands

The first layer is rebuilt: failed reads, memory per part, dates from evidence, bursts as one event. In tests shaped like that week, old pages no longer come back as new, and every rule that sets an item aside is counted in the run's record, so I can see what was left out and why. The other layers are written up and queued, each with its own replay tests to come.

This is a note about method, not a claim about outcomes. I'll add what changed once the rest has run for a few weeks.

What I'd tell myself a week ago

Your best test is the job done well by hand. Diff against it.
Make failure visible all the way up. Silence is the most expensive output a system has.
Keep state at the grain where things succeed or fail.
Write down what code may decide and what a person must judge.
Replay real failures as tests, through the production entry point.