Error handling and tests

When an AI builds automations for you, two questions matter more than anything: what happens when something breaks, and how do you know a change works before it reaches real customers?

When something breaks

Any step can fail: an API is down, a model returns nonsense. I added error paths, so a failure can be routed somewhere useful, like a Slack alert with the details, instead of just stopping. Runs can retry themselves, and a failed run can even trigger an agent that looks at what went wrong and proposes a fix.

Proving it works

This is one of the features I am proudest of. Test layers let you test a pipeline, including its AI steps, without touching anything real.

You feed in a pretend ticket or Slack message, the real logic runs, and anything that would send an email or write data is replaced by a stand-in that just records what would have happened. Then you check the result:

• for normal steps, that the right thing came out;
• for AI steps, that the agent used the right tools and reached a sensible answer, judged by another AI.

A test refuses to run if something real is still wired in, so a test can never message a customer. And because tests follow the real pipeline, changing the pipeline updates its tests.

The best part is what this does for Chip: it writes tests for what it builds, runs them, and keeps fixing until they pass. In the screenshot a test had just caught a real bug in a pipeline Chip was working on.