12 Questions to Ask Before You Buy Any AI Tool

I've sat through a lot of AI sales demos. They're always impressive. The demo data is always clean, the questions are always the ones the tool is great at, and the presenter never has a bad day.

Your organization isn't a demo.

Before you sign anything, ask these twelve questions. I've grouped them into four areas. For each, I'll tell you what a good answer sounds like and what should make you walk away.

Your data

1. Do you use our data to train or improve your models? Good answer: "No, not by default. Here's the contract language." Or a clear opt-out that's in writing. Walk away if: They can't give you a straight answer, or it's only in a marketing FAQ, not the contract.

2. Where is our data stored, and who can access it? Good answer: A specific region or country, named subprocessors, role-based access, and access logging. Walk away if: "It's in the cloud" is the whole answer.

3. How long do you keep our data, and can we delete it? Good answer: Defined retention periods, and a way to delete everything when you leave, with written confirmation. Walk away if: Data is kept "as long as necessary," with no deletion process.

Accuracy and fairness

4. How do you measure accuracy, and on what data? Good answer: Specific metrics, on data similar to yours, with known failure cases they're willing to tell you about. Walk away if: A single headline number like "98% accurate" with no context. Accurate at what, on whose data?

Here's a technical point worth understanding. A model can be highly accurate overall and still fail badly on a specific group. If 95% of the test data looks like one population, the model can score great while failing the other 5%. Always ask for performance broken down by the groups you serve.

5. How do you test for bias? Good answer: Regular testing across groups like race, gender, age, disability and language, with results they'll share, and a process for fixing problems. Walk away if: "Our AI is objective." No system trained on human data is automatically objective.

6. What happens when it's wrong? Good answer: Clear ways for a human to review, override and correct outputs, and a way to report errors back. Walk away if: There's no way to override it.

Control and security

7. Can it take actions on its own? How do we limit them? Good answer: Granular permissions, approval steps for consequential actions, and least-privilege access by default. Walk away if: It needs broad admin access to "work properly."

8. How fast can we shut it off? Good answer: A clear, documented way to revoke access immediately, which you control, without calling support. Walk away if: They seem surprised by the question.

This one matters more every month. In October, the European Commission said more than 20 loss-of-control incidents involving AI agents had been documented this year across major labs and evaluators (Unite.AI). I wrote a whole guide to this question in Who Holds the Kill Switch?

9. How do you protect against prompt injection and data leaks? Good answer: They know what prompt injection is (hidden instructions in content that trick an AI into doing something it shouldn't), and can describe specific defenses and testing. Walk away if: Blank stare.

10. Have you had security incidents, and how would you notify us? Good answer: Honesty about past incidents, a defined notification timeline in the contract, and a named contact. Walk away if: "We've never had any issues," with no notification terms in writing.

Cost and exit

11. What does it really cost at our scale? Good answer: Clear pricing, including usage-based costs, with an estimate for your volume and what happens if you go over. Walk away if: Pricing is "custom" and they won't estimate. AI costs can scale with usage in ways that surprise people.

12. How do we leave? Good answer: You can export your data in a standard format, there's a reasonable notice period, and they delete your data after you go. Walk away if: Your data, prompts or workflows are locked into their format.

The step most people skip: a real pilot

Never sign a long contract based on a demo. Ask for a short paid or free pilot, using your own data, on one specific task. Measure time saved, errors and how your team feels, before and after. I walk through exactly how in A 30-Day Pilot That Won't Waste Your Money.

And before any of that, make sure your data is ready. The best AI tool in the world can't fix a broken foundation. That's the whole idea behind the Data House.

Print this

Bring these twelve questions to your next vendor meeting. Write down the answers. Share them with your team, and with whoever has to approve the purchase. If a vendor won't answer them in writing, you've learned what you needed to know.

Pour me another: What question do you wish you'd asked before buying a piece of software? Tell me through the contact form.

Sip slow, stay curious.D