Red doesn't always mean broken

One evening, two things went red in quick succession. Neither was caused by the code I'd just written. Both looked exactly like it was.

Failure one: the deploy that broke on a font

A deploy failed with this, inside the framework, not inside my code:

An error occurred in `next/font`.
TypeError: Cannot read properties of null (reading '1')
    at .../@next/font/dist/google/loader.js

The change being deployed didn't touch fonts or the layout. The real cause is a build-time dependency most people forget: the framework downloads Google Fonts during the build. When that download comes back as something the loader can't parse, a regex finds nothing, and the build dies on null[1].

The first time, I called it a flake and moved on. It failed again about an hour later, on a different change. That made it a pattern, and a pattern needs a real fix, not a retry.

The fix: stop fetching fonts at build time. I downloaded the exact files the build was fetching (Latin subset, woff2; eight files, about 400 KB) into the repo and switched to loading them locally. Same faces, same variable axes, same CSS variables. The site looks identical. The build no longer depends on a network call it doesn't control.

tsconst inter = localFont({ src: '../fonts/Inter-latin-opsz-wght.woff2', weight: '100 900', variable: '--font-inter' });

The lesson is general: anything your build fetches is a dependency that can fail. List those fetches and decide which ones you'd rather own.

Failure two: every check failed in two seconds

Then every CI check on every pull request went red. Typecheck, build, tests, secret scan, all of them failed after two or three seconds.

Two seconds is the clue. Real failures take minutes: install dependencies, compile, run tests. Failing that fast means the job never started. My account had used its included CI minutes for the month, and the spending budget was set to $0, so the platform refused to start new jobs. Nothing was wrong with the code.

How I handled it:

1. Prove it, don't assume it. I ran the same four checks locally on the branch: the font-size lint, typecheck plus production build plus bundle budget, the test suite, and the secret scan. All passed. The real deploy (built by the hosting platform, not CI) passed too.
2. A small budget, capped. $10 with "stop when reached" covers the rest of the month and can't run away.
3. Make CI cheaper. This was the real win.

Making CI do less

Every pull request ran four jobs, each billed in whole minutes, and ran them all again after merge. A change that only touched the backend still built the whole website, and vice versa.

The workflow is now three, each triggered only by the files it cares about:

• Web: runs only when the web app changes. The tiny font-size lint, a one-second grep that used to bill a whole minute of its own, now runs inside this job.
• Worker: runs only when the backend changes, with a package cache.
• Secret scan: runs on every pull request, but not again on merge, because the merged code is what the pull request already scanned.

A docs-only change now runs only the secret scan. Same checks as before, compared step by step, with far fewer minutes.

The habit I'm keeping

Read the shape of the failure before reading the error.

• Fails in two seconds → it didn't run. Look at quotas, permissions, billing.
• Fails in the framework, in code you didn't change → look at what the build fetches or depends on.
• Fails the same way twice on unrelated changes → it isn't a flake.
• Fails in your own code → now read the stack trace.

And when CI can't run, run the checks yourself and write down the results where reviewers will see them. "Green locally, here's the output" beats "it's probably fine."