Story, Gate, Ladder, Ledger

Observation

When an agent says something is done, I learned not to take it literally. Once, an automated check reported a new part of a screen as tested, 2 out of 2, and that part had never even appeared on the screen. I started calling this “looks done but isn’t”, and it became the first thing I design against.

Principle

No story, no measure; no measure, no merge.

Loopstr, CLAUDE.md §1

Mechanism

So every change in Loopstr starts as a short story with a result you can measure. I commit the story first, on its own, while it still fails. Then the code. If the code comes first, CI rejects the branch. This still works when the branch gets squashed into one commit, because the story leaves a line in the commit message.

Evidence

The week I turned the gate on, 463 stories were written against 270 merged changes. A lot of untested work suddenly had to be measured. Since then the two numbers move together.

Open question

I don’t know yet if writing the story first means fewer bugs reach people, or only that I see them earlier. I need a comparison to answer that.

Observation

In the beginning my agents asked permission for everything, and I started clicking yes without reading. My gate code calls this “approval fatigue / rubber-stamping”. Then I realized most of what agents do for me (research, drafts, designs, builds) only creates files I can throw away. Only a few steps actually leave the system: publishing, deploying, creating a database.

Principle

The gate belongs where an action is hard to undo, not on every step.

Loopstr, GATE-AUTONOMY-LADDER.md

Mechanism

So each kind of step now says whether it can be undone. If it can, the agent moves on by itself when it is confident enough, 0.75 or higher. If there is no confidence signal, it waits. Verify, ship and app always wait for me. An automatic run has to stop before the first step it can’t take back, and a project can make its gates stricter but never looser. In Looper, five kinds of tasks always need a person, no matter how much trust the agent has earned.

Evidence

Open question

This rule changes where runs stop. I haven’t shown yet that it prevents real damage. I want to test it by planting risky actions in a sandbox.

Observation

A test that can’t fail looks exactly like a test that passes. In Loopstr, one function was supposed to say “unknown” and returned a made-up confidence of 0.5 instead. It stayed there for months and let empty runs pass their own gate. Another time I compared what agents said their tests did with real runs: 2 out of 5 had reported results that never happened, even though the brief told them not to.

Principle

A check that cannot fail is not a check.

Loopstr, docs/LADDER.md; agents-discipline

Mechanism

Every time I closed one way a check could fool me, I found the next one. I wrote them down in order, thirteen so far, each with the tool that catches it. Break the code on purpose and see if the test goes red. Run two cases that have to give different answers. Don’t show a claim as done until it points to a source. If a value is unknown, leave it empty, never fill it with a good-looking guess. Four of the rungs still don’t have a tool, and the ladder says that too.

Evidence

Open question

I don’t know if thirteen is the full list or if it keeps growing. Trying the rungs on tests that agents wrote for other people’s projects would tell me.

Observation

My first reaction to an agent’s mistake was to add another rule to its instructions. Loopstr’s rules file went from 392 words to 12,857 in nineteen days, and the same mistakes kept coming back. The lessons ledger started on the day I wrote “i thought we were but we werent.” A study published this year found something similar: these instruction files didn’t make agents more successful and made them about 20% more expensive.

Principle

A lesson gets exactly one home, in this order: CI gate, hook, skill, docs, prose.

Loopstr, CLAUDE.md §12

Mechanism

So now, at the end of each session, a hook saves the corrections I made, and the build won’t pass while any of them is unsorted. Each lesson goes to the strongest place it can live: an automatic check if possible, plain text only if nothing else works. The ledger also records what each of my decisions cost, and whether I gave it, steered it or left it to the agent. And my own words stay the way I wrote them. I didn’t want agents to keep “filtering my thoughts like a phone game.”

Evidence

Open question

The cost-per-decision part is built, but it barely has any data yet. Next I want to run it for a few weeks and see how much of the cost comes from my own steering.