No. 20

12 Questions to Ask Before You Buy Any AI Tool

AI vendors promise the moon. Before you sign, ask these twelve questions about your data, accuracy, security, bias, cost and exit. Includes the answers that should make you walk away.

The Tea the short version

  • Most AI risk comes from the contract, not the code. Ask before you sign.
  • The big four: what happens to your data, how accurate it is, who can shut it off, and how you leave.
  • Vague answers are answers. 'It's proprietary' is not good enough for questions about your data.
  • Run a small pilot with your own data before any long contract.

I've sat through a lot of AI sales demos. They're always impressive. The demo data is always clean, the questions are always the ones the tool is great at, and the presenter never has a bad day.

Your organization isn't a demo.

Before you sign anything, ask these twelve questions. I've grouped them into four areas. For each, I'll tell you what a good answer sounds like and what should make you walk away.

Your data

1. Do you use our data to train or improve your models? Good answer: "No, not by default. Here's the contract language." Or a clear opt-out that's in writing. Walk away if: They can't give you a straight answer, or it's only in a marketing FAQ, not the contract.

2. Where is our data stored, and who can access it? Good answer: A specific region or country, named subprocessors, role-based access, and access logging. Walk away if: "It's in the cloud" is the whole answer.

3. How long do you keep our data, and can we delete it? Good answer: Defined retention periods, and a way to delete everything when you leave, with written confirmation. Walk away if: Data is kept "as long as necessary," with no deletion process.

Accuracy and fairness

4. How do you measure accuracy, and on what data? Good answer: Specific metrics, on data similar to yours, with known failure cases they're willing to tell you about. Walk away if: A single headline number like "98% accurate" with no context. Accurate at what, on whose data?

Here's a technical point worth understanding. A model can be highly accurate overall and still fail badly on a specific group. If 95% of the test data looks like one population, the model can score great while failing the other 5%. Always ask for performance broken down by the groups you serve.

5. How do you test for bias? Good answer: Regular testing across groups like race, gender, age, disability and language, with results they'll share, and a process for fixing problems. Walk away if: "Our AI is objective." No system trained on human data is automatically objective.

6. What happens when it's wrong? Good answer: Clear ways for a human to review, override and correct outputs, and a way to report errors back. Walk away if: There's no way to override it.

Control and security

7. Can it take actions on its own? How do we limit them? Good answer: Granular permissions, approval steps for consequential actions, and least-privilege access by default. Walk away if: It needs broad admin access to "work properly."

8. How fast can we shut it off? Good answer: A clear, documented way to revoke access immediately, which you control, without calling support. Walk away if: They seem surprised by the question.

This one matters more every month. In October, the European Commission said more than 20 loss-of-control incidents involving AI agents had been documented this year across major labs and evaluators (Unite.AI). I wrote a whole guide to this question in Who Holds the Kill Switch?

9. How do you protect against prompt injection and data leaks? Good answer: They know what prompt injection is (hidden instructions in content that trick an AI into doing something it shouldn't), and can describe specific defenses and testing. Walk away if: Blank stare.

10. Have you had security incidents, and how would you notify us? Good answer: Honesty about past incidents, a defined notification timeline in the contract, and a named contact. Walk away if: "We've never had any issues," with no notification terms in writing.

Cost and exit

11. What does it really cost at our scale? Good answer: Clear pricing, including usage-based costs, with an estimate for your volume and what happens if you go over. Walk away if: Pricing is "custom" and they won't estimate. AI costs can scale with usage in ways that surprise people.

12. How do we leave? Good answer: You can export your data in a standard format, there's a reasonable notice period, and they delete your data after you go. Walk away if: Your data, prompts or workflows are locked into their format.

The step most people skip: a real pilot

Never sign a long contract based on a demo. Ask for a short paid or free pilot, using your own data, on one specific task. Measure time saved, errors and how your team feels, before and after. I walk through exactly how in A 30-Day Pilot That Won't Waste Your Money.

And before any of that, make sure your data is ready. The best AI tool in the world can't fix a broken foundation. That's the whole idea behind the Data House.

Bring these twelve questions to your next vendor meeting. Write down the answers. Share them with your team, and with whoever has to approve the purchase. If a vendor won't answer them in writing, you've learned what you needed to know.

How to score a vendor's answers

Bring a simple scoring sheet to the meeting. For each of the twelve questions, give the answer a score:

  • 2 = clear answer, in writing or in the contract
  • 1 = verbal answer, or "we'll get back to you"
  • 0 = no answer, vague answer, or a red flag

The top score is 24. In my experience:

  • 20 or higher: worth a pilot.
  • 14 to 19: ask for written follow-up on the 0s and 1s before going further.
  • Below 14: walk away, or come back in six months.

Scoring keeps you honest. Demos are designed to make you feel good. A number on paper is much harder to argue with, especially when you have to explain the decision to a board or a business partner.

Contract clauses worth asking for

Many of the twelve answers only protect you if they end up in the contract. These are the clauses I ask organizations to look for, or request:

  • No training on customer data without separate, written permission.
  • Data deletion within a set number of days after the contract ends, with written confirmation.
  • Breach notification within a set number of hours or days, with a named contact.
  • Data export in a standard format like CSV or JSON, at no extra charge.
  • Subprocessor list, with notice before they add a new one.
  • Price protection: a cap on increases at renewal.

You may not get every one, especially with a small contract. But asking tells the vendor you're paying attention, and the answers tell you how they treat customers.

Red flags in the demo itself

Watch the demo as closely as you listen to the answers.

  • They only use their own sample data. Ask them to run it on a sample of yours, with private details removed.
  • They won't show a mistake. Every system makes mistakes. Ask, "Show me what happens when it gets something wrong."
  • The person demoing can't answer technical questions and there's no one on the call who can. You'll be dealing with that same gap after you sign.
  • Pressure to sign quickly. "This price is only good this week" is a sales tactic, not a reason.

Questions for nonprofits and public agencies specifically

If you serve vulnerable people or handle public records, add these:

  • Accessibility: Does the tool work with screen readers and keyboard-only navigation? Ask for its accessibility documentation.
  • Languages: How well does it perform in the languages your community speaks, not just English?
  • Records: If you're a public agency, can you retrieve what the tool produced if there's a public records request?
  • Community impact: Who could be harmed if this tool is wrong, and how would you know?

If your data foundation isn't ready, none of these answers will save the project. Start with Get Your Data House in Order Before You Buy AI.

Questions people ask

What should I ask an AI vendor about my data?

Ask whether they use your data to train or improve their models, and get the answer in the contract. Ask where data is stored and who can access it, how long they keep it, and how you can delete everything when you leave.

How do I know if an AI tool is accurate?

Ask for specific accuracy measures on data similar to yours, the known failure cases, and performance broken down by the groups you serve. A model can be highly accurate overall and still fail badly for a smaller group.

What are red flags in an AI sales pitch?

A single headline number like “98% accurate” with no context, “our AI is objective,” no way to override the tool, a need for broad admin access, surprise when you ask how to shut it off, and pricing that's “custom” with no estimate.

Should I run a pilot before buying an AI tool?

Yes. Never sign a long contract based on a demo. Ask for a short pilot using your own data on one specific task, and measure time saved, errors and how your team feels before and after.

Pour me another: What question do you wish you'd asked before buying a piece of software? Tell me through the contact form.

Sip slow, stay curious.D

Receipts 1 sources cited in this post
  1. Unite.AI unite.ai
Dei’Marlon “D” Scisney

Your host

Dei’Marlon “D” Scisney

Founder & Principal Data Engineer, The Data Guys™

Data engineer, teacher and civic technologist. Vice-Chair of Seattle’s Community Technology Advisory Board. Raised by his grandmother in Evansville, Indiana; building community technology centers in Seattle’s Central District and back home in Evansville.

I am not a lawyer, and nothing here is legal advice. Facts were current when this was written, and every claim links to its source. Spot something that has changed? Tell me and I’ll update it.

Another cup?

Keep sipping.

Pour me another

Get the tea before everyone else.

Fresh posts on data, AI, privacy and the people tech forgets, plus The Tea Report, straight to your inbox. Written by me, never sold, never shared.

Free. No spam, ever. Unsubscribe in one click. Prefer LinkedIn? Subscribe there.