Creed

Software factories are a contested topic, with some teams claiming completely lights-off factories — where nobody reads or manages a single line of code. The other side says that lights-off does not hold on complex, scaled, production codebases.

Punchcard is a financial audit platform ingesting and checking thousands of documents every day in live audits, with a large breadth of features and systems to maintain. All said, the constraints as the platform relentlessly grew were very real: Nathan Turnbow and I had a team 1–5% the size of the incumbents, in an industry where reliability meant everything, while also seeking to spend every minute with customers working on novel problems and keeping a lead on innovation in our space.

It didn’t take long for the idea of an internal engineering agent to germinate and sprout in my mind, before the “factory” term was coined, but it was certainly that same shape of solution.

The two ambitious targets were:

• No more user-reported bugs (zero!)
• A horizontally scalable, peer-level engineer who could be trusted with end-to-end ownership of finding bugs and fixing them, unsupervised

That said, having worked with AI coding agents for a while, I knew the risk of an “AI-slop cannon” at scale:

• Heavy token burn for marginal insight
• Half-finished or shortsighted fixes
• Risky code around customer data and hot paths
• Accidentally optimizing for the wrong reward
• Maintenance, which is always easy to overlook

The earliest version was dubbed “Exterminator” — the ultimatum for bugs rather than another bug bot: squash them before they hit a user. We later renamed it Creed, after the iconic head of QA at Dunder Mifflin.

On the spectrum of software factory zealots vs. skeptics, we have found great results somewhere in between — very often lights-out, but built so the factory stays sustainable as the product and customer base grow.

These are the areas I see as delivering the key systemic balances:

Three loops, not one agent

Rather than a single orchestrator, Creed is a distributed graph of ~40 actions (each a role, incentive, and trigger), grouped into three self-contained loops.

• Map maintains the working model of the product and turns it into missions.
• Verify checks those missions against reality (browser + what prod is already flagging).
• Fix turns a verified finding into a merged, re-tested change.

The jobs themselves are each self-contained. A cron, a label, or a deploy triggers and enqueues work; the worker writes a persisted artifact and exits. Subsequent actions read from the same memory and global state. Orchestrated entirely through GitHub Actions and self-hosted runners on AWS (before cron jobs and Cloud Agents started shipping as features with tools like Cursor and Linear).

High signal or bust

This will be a recurring theme in more than one area of Creed, but token burn for marginal insight tends to produce a lot of noise. Low signal-to-noise output just gets ignored, and is worse than no signal at all.

When it comes to proactively discovering bugs, it turns out Playwright-based browser testing runs can be flaky, for various reasons, causing false reports. When we required repeatability across two occurrences, observed false positives were close to zero.

Additionally, an LLM’s verbalized confidence is not a score you can reliably threshold like an object detection model, so we treat a bug report as a binary: is this reproducible, or not?

You can’t tell the story of Moby Dick from blueprints

Inevitably, a system like this requires intelligent awareness of real user journeys through the product in order to know what to prioritize and test. A naive approach to building that knowledge is asking an LLM to infer user journeys and product intent from static code analysis. But reading a React component tree to reconstruct intent is not unlike trying to tell someone the story of Moby Dick by looking at Captain Ahab’s ship blueprints.

Fortunately, we already had an important building block available. Self-maintaining, user-facing product documentation, built for our platform changelog and as queryable context for our generalist platform agent, CoAudit — framed from a user perspective, derived from PR and commit descriptions (which coding agents have done wonders for).

Later we paired that with more structured maps that deterministically resolve dependencies and seed fixtures from a lexical mission definition — one less step burning token budget every session.

Nobody grades their own homework

Show me the incentive and I will show you the outcome.

To stay unsupervised and long-running (meaning, weeks to months without intervention), it needed both the right incentives and no way to game them (the way frontier models sometimes reward-hack their evals). An easy gaming example: give it “drive reported bugs to zero” and it’s not out of the question that it authors its own softball tests that don’t actually surface bugs, or worse, deletes whole features in the interest of stability (Tron Legacy has a whole movie plot around this).

As such, approval/merge was the last purely human-in-the-loop step.

After a month or two of reviewing every Creed PR, I had a sufficiently representative labeled set to run an experiment around auto-approval. In the spirit of Karpathy’s autoresearch approach: we ran an optimization loop on a single code review prompt where each new prompt refinement was only kept if

1. it auto-approved a higher percentage of valid PRs than the previous version, and
2. did not incorrectly approve any reverted or closed PRs in the labeled set (an “escape”)

The core set of roles required by Creed to operate autonomously is shown in the following matrix, which represents full coverage as a system, but no ability for any one agent to escape and game its own incentives.

These roles in aggregate ensure that both progress and sustainability are maximized equally.

Very important to note here: our software runs in live financial audits, with sensitive client data. Auth, migrations, infra, database mutations are areas that are marked to universally require human approval (never automatic).

I am a big fan of the concept of backpressure: the systems and code that ground the agent and prevent sloppy implementation, code bloat, and generally keep agents on the efficient path — lint, type checks, dead-code scans, tests, proof-of-work, adversarial review, and so on.

Proof-of-work was a great example of backpressure at merge: a screenshot empirically proved to human and agent alike that the app compiled, the full stack still works, and the UI didn’t regress — multiple questions at a glance. From there, a reviewer can immediately drill the known dangerous diffs, if any, without needing to check out the feature branch and boot up a new local instance themselves.

The scarce, finite resource being protected throughout was human attention. The visualization below shows how ~59% of all Creed PRs were found, fixed, and approved without any human intervention.

Something has to fix the rails

Maintenance is the one that’s often easy to underestimate. In this case, we knew that a distributed system like this would have its own share of paper cuts. Flaky infra, poisoned or stuck runners, one queue outpacing another and causing imbalance, and so on.

A nightly agent looks at the north-star fleet metrics — PRs merged, missions run, PRs reverted, queue health — and can author fixes to the rails.

Results

This is coverage that let us both prevent and react to bugs. Though I don’t have exact support-case numbers, the shift has been noticeable. From my own anecdotal experience I’d estimate we’ve fielded close to a fifth of the bug reports while the customer base and activity grew about 5–6× over the same period.

Was it just better models? Those should also have helped humans ship fewer bugs. In our own repo, bugs caught in code review did not drop by anything significant over the same period.

The mark of an unsustainable system is reported issues climbing with time, and requiring increasingly more resources to untangle the mess.

The future-proof part of this software factory architecture is that for every stride model advancements make for coding reliability, the only meaningful adjustment needed is relaxing the auto-approve gates accordingly. The same efficiencies and guarantees from distributed roles, signal-to-noise prioritization, and maintained knowledge base remain reusable.