Five Incidents, and None of Them Were the Migration
Surfaced is a service I run that keeps local businesses findable. It drafts a reply to every new Google review in the owner’s voice, writes posts for their Google Business Profile, and writes and improves pages on their website. Nothing goes live until the owner approves it.
In late August I moved it off n8n, the workflow tool it had grown up on, and onto Forge, the agent runtime I build everything else on. Its clients didn’t have to change anything, and there’s a plain-language version of what changed for them on the Surfaced blog.
This is the other version, for anyone who builds systems like this or pays someone to run one. It’s mostly about what went wrong.
Why move at all
The case for it arrived in the first days of the work. Moving the first data feeds turned up one that had stopped working without anything reporting it. That’s the real risk in a service like this. It earns its keep in the weeks nobody is watching, so a job that quietly stops is the worst kind of failure.
The cause was structural. The work was being started from three places that couldn’t see each other: n8n’s timers, two background threads inside the application, and a separate listener on the server. Each did its job. None of them knew whether the others had done theirs.
Where it landed
The migration took three weeks, 106 commits and 15 merge requests. As of the retrospective I wrote on September 18:
| Before | After | |
|---|---|---|
| Workflows doing work in n8n | 11 | 0 |
| Scheduler threads inside the application | 2 | 0 |
| Agents, each with one job | 0 | 13 |
| Schedules, each with its state on disk | 0 | 15 |
| Tests | 37 | 229 |
| Rank-tracking spend | $1.74 a week | $0.42 a week |
The number worth dwelling on is the zero. There is now one scheduler, and every run leaves a file behind. So does a run that should have happened and didn’t. Between August 27 and September 18, agents ran 197 tasks. Two failed, and both produced a fix with a test. n8n itself was shut down for good on September 23.
Two kinds of work, one rule
The service does two quite different things, and I’ve learned that owners feel very differently about them.
The Google Business Profile. New reviews are checked three times a week, and each one gets a reply drafted in the owner’s voice. Posts are drafted once a month. When the owner approves one, it’s live on Google within a couple of seconds. Before posting a review reply, the system checks Google for a reply the owner wrote themselves, and if there is one, it records theirs and posts nothing.
The website. When an owner approves an idea for a new page, the page is written, built and put on a real preview link. The owner checks the finished page and approves it a second time before it goes live. The first time I ran that loop end to end on a live client site, from approval to a finished preview, it took 2 minutes and 23 seconds.
In my experience, owners who are wary of anyone touching their website are often happy to let a system draft their review replies. The profile is lower stakes and more visible, and it’s where many of them would rather start. But the same rule governs both: the last step before anything goes out to Google or a website reads the approval status itself, and refuses anything that isn’t approved. Approval isn’t something the agent is asked to respect. It’s something the publishing tool checks.
How the move was done
Four rules, and I’d use all of them again.
- Switch off, never delete. Every old workflow was deactivated, not removed, until the new system had run on its own for a while. Nothing ever needed rolling back, but every cutover could have been reversed the same day.
- Observed, not tested. Nothing was switched off until its replacement had been seen doing the job on a live client’s profile or site, with the result written to the record.
- Every fix carries its incident. Migrations, tests and comments name the date and the failure they exist for. That’s why the retrospective could be assembled from the repository rather than from memory.
- Every site change goes to a preview first. It costs nothing. Once, it caught a stale branch that would have reverted a week of live client content.
What broke
Nothing broke during a cutover. Five things broke afterwards. Each was either a defect the move exposed, or one the move introduced that a later schedule tripped over days afterwards.
The planner was silently dead for nine days. The two schedules that propose new website pages were set up with a future start time, fired early anyway, and claimed September’s slot. When the real September run came, it was skipped as a duplicate. A skip isn’t a failure, so nothing reported anything. The nine lost days weren’t the real damage. That was seven stale proposals, aimed at a live client site, that still looked current to anyone about to approve one.
One comma refused 700 keywords. The keyword data provider rejects an entire batch if one phrase in it is malformed. A comma killed 700 keywords. A stray character killed 700 more. Worse, the code that saved the results then recorded 1,392 keywords as having zero searches. Validation in front of a paid service that charges per item has to be a list of what’s allowed. And a test that checks nothing was parsed doesn’t prove nothing was written.
The rank tracker billed twice. The tool call timed out after ten minutes. The work took about twenty-five. The server didn’t stop when the call was abandoned, and the agent retried a run that was still spending money. It had happened once before and been recorded as a success. An agent must never retry a tool that spends money. If retries are needed, they belong inside the tool, at the level of the thing being billed.
One keyword cost a week of Search Console data. Two phrases that differed only in capital letters turned into the same internal key. Saving one of them hit a uniqueness rule, and the error took the whole batch of 100 rows down with it. Only 129 of 608 rows for that week arrived, and a weekly briefing reported an 86% traffic drop that never happened. The arithmetic was correct. It was correct about a week that was four-fifths missing.
The staleness alert was the threshold. Rank tracking moved from twice a week to weekly, but the rule for “this feed is stale” still said four days. From then on the feed was reported stale every week, with nothing wrong behind it.
The same mistake, three times
Three of those five are one defect in three different places: a loop over independent items, where one item’s failure destroys all the others. One bad phrase refused a batch of 700. One refusal re-ran every keyword for every client. One colliding key took down a batch of 100.
I wrote the rule down after the second one: one item must not be able to end the batch. It fired again three days later, in a part of the system nobody had thought to check against it. Writing a lesson down is not the same as applying it. The useful follow-up isn’t a note, it’s a sweep for every other loop of that shape.
The monitor was confidently wrong
There’s a monitoring agent whose job is to notice when a feed goes stale and say why. Twice in six days it was wrong, and both times it was wrong with confidence rather than silent. Once it said three schedules that fix themselves needed manual intervention. Once, seeing that stale feed next to an unrelated warning, it invented a cause linking the two and told me the wrong thing to fix.
A monitor that overstates spends its credibility, and credibility is the only thing a monitor produces. It was built early, as a prompt with no checks behind it, and never revisited while everything it watched was rewritten underneath it. It deserved the same rigour as the code, and it didn’t get it.
What I’d do differently
- Run the schedule after changing what it touches. Four of the five incidents surfaced days after the change that caused them, because the job that would reveal them ran weekly. A manual run straight after the change would have caught the key collision a week earlier.
- Keep a threshold next to the schedule it describes. The staleness rule and the schedule lived in two different files, and nothing connected them. There’s now a test that derives one from the other. The first time it ran, it found a second mismatch nobody knew about.
- Treat the monitor as code. Not as a prompt you write once.
What this means if you’re buying, not building
If someone runs automation on your behalf, whether that’s your reviews, your marketing or your operations, you can’t see any of the above. What you can do is ask:
- What happens when a job doesn’t run? Is there a record that says so, or does it wait for you to notice that the phone rings less?
- Can anything go out without your approval, and where is that enforced? A promise in a contract, an instruction to the model, or a check in the step that publishes?
- When something breaks, can they show you what happened? Or do they have to reconstruct it from memory?
That last one is the thread running through everything I build. I wrote about it in Keep the Receipts, Swap the Model: if you can’t look up what the system did and why, you’re trusting it, not running it.
If your business has automations nobody has looked at since they were set up, a walkthrough is a good place to find out which ones are still doing their job. Start here.