When I wrote about Carson, the household agent that sends my partner and me a message about tomorrow every evening, the fair objection was obvious: couldn’t I just do that with a bot I subscribe to?

Some of it, yes. A general-purpose assistant can read school email and summarise it. What it can’t give you is two things I now think matter more than any feature. The first is a record of everything the agent did and why. The second is the freedom to change the model underneath it, on my schedule, without finding out from the results.

Both sound like engineering hygiene. They are really about who is in charge.

The receipts

Carson keeps a record of everything it does, and none of it was added for this post. It is how the system works.

Every item it pulls out of a school message is stored with the exact words it quoted, how confident the model was, whether the item went into a brief or was held back, the reason, and which version of the model and instructions produced it. So far that is 106 items. Ninety went into a brief. Sixteen were held back as questions for a parent instead, the kind of item where a date could mean two different days or the quote didn’t check out.

Every brief is stored with who it went to, when it was sent, and whether the send worked. Underneath that, the runtime it runs on records every run as an append-only transcript: what the agent was asked, which tools it called, what came back. Nothing in that record is edited after the fact.

Here is why that matters. One evening the brief went to me and not to my partner. I didn’t have to guess what happened. The send record showed the brief had gone out with a recipient setting it should never have touched. The agent’s run showed where that came from: it had filled in an optional field on the send tool, one I had added for my own testing. A model offered an optional field will usually fill it in. I removed the option entirely, and wrote the rule down so I wouldn’t make that mistake again.

Without the record, all I’d have known was that my partner didn’t get a message. I could have blamed the email provider, or the model having an off night. With it, I knew which tool, which field and which run, and the fix was a design change rather than a hope.

The same goes for a business. When someone asks why the system quoted a customer the wrong lead time, or why an invoice was flagged, “the AI decided” is not an answer. A record is. If you can’t look up what the agent saw and did, you are trusting it, not running it.

I’ll be honest about where my records stop. They show what happened. They don’t yet tie every conclusion to the specific evidence it rests on, so that “the brief said Thursday” points straight to the sentence that said Thursday. Carson does that for school items because it stores the quote. Doing it in general is designed but not built.

The swappable model

Carson started on one company’s model. Eleven days in, I moved its extraction, the part that reads each school message, to a different company’s model. For a system that tells two parents what their child needs tomorrow, that is not a small change.

The switch itself was mostly plumbing. One line of configuration. A new way of making the request, so the new model’s answers are forced into the exact shape the code expects. Handling for the case where the model declines to answer. Removing a setting the new model rejects.

Knowing the switch was safe was the real work, and that work had already been done. Before Carson ever ran on its own, I built a test set: synthetic school messages with known answers, built from the awkward patterns real ones have, plus real messages with answers written by hand. 141 expected items. Each case runs three times, because a model doesn’t give the same answer every time. The bar is zero hard failures: nothing missed, nothing invented, nothing that should have been a question presented as fact.

I ran the same test against the new model before switching it on. It disagreed with my answer key exactly once. A notice about an early release day: my expected answer filed it as information, and the new model filed it as something to act on. I thought about it and the new model was right. An early pickup is something you have to arrange. I changed the expected answer, ran it again, and got 141 of 141.

That moment is the point. The test isn’t there to rubber-stamp a new model. It is where a disagreement surfaces, and a person decides who is right before it reaches anyone’s evening.

Why this matters more than features

With a subscription bot, the vendor chooses the model, and changes it when they decide to. Sometimes it gets better. Sometimes the thing you relied on quietly starts behaving differently, and you learn about it from a customer, or from a child standing at school in the wrong shirt.

With your own system and your own test set, you choose when to change the model. You get to find out what changed before it matters, and you can move to whichever model is best or cheapest for the job this quarter, without starting again.

Neither of these is about AI being clever. Both are about you staying in charge of something that acts on your behalf. That is what I build for businesses: your rules enforced by the system rather than requested of the model, a record of everything it does, and a model you can replace.

Carson runs on Forge, the agent runtime behind the systems I build for clients. If you’re wondering what your business would want a record of, that’s a good place for a walkthrough to start.