I’m building Friction Fixer, a team of AI agents that studies why visitors leave a site, ships a fix as a pull request, and undoes it if the numbers don’t improve. The agents were the easy part. The hard part was deciding when to trust the result.
The rule
A change is kept only if bounce rate falls by at least 5 percentage points. Otherwise a second pull request reverts it.
What I learned
Waiting time matters more than the agent. The default wait before checking results is 7 days, to match the 7-day baseline. You can set it as low as 1 hour for a demo, but one hour of traffic is unlikely to move bounce rate by 5 points. So a demo at the shortest wait should expect an UNDO. That isn’t the agent failing, it’s not enough data.
Mock data proves the loop, not the fix. Real traffic is too slow for a 48-hour hackathon, so I added mock visitors. But in mock mode I pick whether the outcome is “improved” or “worse.” That shows the whole loop works end to end (plan, pull request, merge, verdict, revert). It does not show that any edit really helped a real visitor. I label every mock result as mock.
One change per run. Each change is its own pull request, so when the numbers move I know what caused it, and undoing is a single revert.
What I’d do differently
A run failed late one night with “Failed to create branch: worktree contains unstaged changes,” and I only found out where it broke because every round is one neatlogs trace. Next time I’d add tracing before building the agents, not after.
If you’re building agents that change real things, decide how you’ll know they worked before you let them act.