How to Run an AI Pilot (With a Kill Date Attached)

Most AI pilots don't fail. That would at least be a clear result. They do something worse — they drift. Three months in, the tool is sort of being used by sort of half the team, nobody can say whether it's working, and nobody wants to be the one to kill it because of the time already sunk in. That's not a pilot. That's a subscription you forgot to cancel.

A pilot is supposed to produce a decision: keep it, kill it, or change it — by a date you set before you start. Here's the structure we use to get there in 30 days or fewer.

1. Write down the decision before you write the check

Before any tool gets evaluated, answer one question on paper: what will we know at the end of this that we don't know now, and what will we do about it? If you can't finish that sentence, you don't have a pilot — you have a demo you're paying for.

Good pilot questions are specific and disprovable. “Does this cut our quote turnaround from two days to under four hours?” is a pilot. “Will AI help our sales team?” is a vibe.

2. Set the kill criterion and the kill date — out loud

This is the part everyone skips, and it's the whole point. Before you start, name the number that means “this didn't work” and the date you'll check it. Put both in writing where the team can see them.

The kill date does two jobs. It caps your downside — you can't sink six months into something that was dead at week three. And it changes how people behave during the pilot: a tool with a known expiry gets used with intent, because everyone knows a real evaluation is coming, not an indefinite “let's see.”

A pilot without a kill date isn't an experiment. It's an open-ended expense with optimism attached.

3. Scope it down until it's almost too small

The instinct is to test the tool everywhere so you “really know.” That's exactly how pilots drift. Pick one workflow, one team, one measurable outcome. You're not trying to transform the business in 30 days — you're trying to get one clean signal.

Narrow scope is also the only way to instrument it. If twelve things change at once, you'll never know which one moved the number. One workflow, one before, one after.

4. Measure the baseline first — or you have nothing to compare to

You cannot prove a tool worked if you never wrote down what “before” looked like. Spend the first few days capturing the current state of whatever you're trying to improve: the hours it takes, the error rate, the cost, the turnaround. Boring, and non-negotiable. A pilot with no baseline always concludes “it felt faster” — which is exactly the slide-deck answer we're trying to avoid.

5. Count the cost that isn't on the invoice

The subscription price is the cheap part. The real cost of an AI pilot is your team's time — learning the tool, reworking the workflow around it, cleaning up what it gets wrong. When you tally results, count that. A tool that saves four hours a week but costs six to babysit failed, even if the demo was magic.

When the answer is “not yet”

Here's the part we'll say that a vendor won't: sometimes the honest result of a well-run pilot is don't. Not this tool, or not this problem, or not until your data is in better shape. That's not a failed pilot — that's a pilot that did its job. It saved you from the much larger bill of rolling something out company-wide that was never going to pay off.

The whole point of the kill date is to make “not yet” a cheap, early decision instead of an expensive, late one.

The short version

1. Write down the decision the pilot is supposed to produce.
2. Set the kill criterion and the kill date before you start — out loud.
3. Scope to one workflow, one team, one number.
4. Capture the baseline before you change anything.
5. Count your team's time, not just the subscription.

Run it that way and you get the one thing most AI initiatives never produce: a clear answer, on a date, that you can act on. Keep it, kill it, or fix it — then move on to the next bet with your time and budget intact.