What is AI vendor diligence, and how do you do it without becoming the bottleneck?

An AI vendor sends a deck. The deck claims a proprietary model, shows a customer success story with redacted numbers, runs a demo on dummy data, and quotes twenty-four thousand dollars a year with a twelve-month minimum. You suspect parts of it are padding, and you are probably right. You also need to make a decision in two weeks, the COO has a budget line for AI tooling, and a junior engineer on your team likes the product.

What AI vendor diligence actually is

AI vendor diligence is the structured assessment of a vendor pitch before you sign. It does four things:

1. Strips the marketing. Names what the product actually does in technical terms, separate from how the pitch describes it. “Proprietary AI” becomes “a system prompt over a third-party model with a vector store and a queue UI.”
2. Tests against your use case. A generic assessment tells you little. The question is whether the tool helps with your work.
3. Surfaces the specific red flags. The ones in the pitch in front of you.
4. Produces the questions to put to the vendor in writing. Verbal answers can be qualified later. Written answers cannot.

It ends in a recommendation that is one of three things: Go (the product fits, sign with named conditions), Maybe (the shape is right but a piloted test answers the open questions before the annual commitment), or Stop (the product is the wrong shape, the wrong price, or wrong on data; here are the alternatives).

The ten red flags in a vendor pitch

These are the ten patterns I look for. One alone is rarely a reason to walk away; several together usually are.

1. “Proprietary AI” with no model named. Ask which underlying model, who their contract is with, and what the data retention and training position is.
2. The demo runs on dummy data. Insist on a paid trial with your real data before any annual commitment.
3. The data-storage question is dodged. Where the data lives, in which jurisdiction, and whether it is processed outside Australia are answerable in writing in one sentence.
4. The price is five to ten times raw API cost. That is fine if you are paying for UI, audit trail and support. The vendor should be able to articulate what you are paying for.
5. Twelve-month minimum, no pilot option. The counter-offer is a paid pilot of a month or two at roughly one twelfth of the annual quote.
6. No reference customer at your scale in your industry. Ask for two named contacts running roughly your volume, in roughly your industry, live for more than six months.
7. Accuracy claims with no baseline. “Ninety percent accurate” on what dataset, measured how, by whom? Ask for first-pass accuracy on something like yours, plus the human-in-the-loop edit rate at month one versus month six.
8. The roadmap answers more questions than the product. Score the pitch on what ships today.
9. Cancellation terms are vague. What happens to your data, your prompts, and your audit trail the day you stop paying needs an answer in the contract itself.
10. The seller cannot explain how it works in plain English. If the explanation hides behind jargon twice in a row, the gap is usually in the product.

The free two-page checklist I use to read pitches is on the Insights page. It carries the same ten flags with the question to ask printed next to each one. Take it into the vendor meeting and note which questions get a straight answer.

The wrapper test

A common pattern in AI vendors in 2026: the underlying capability is a system prompt over a third-party model (almost always OpenAI’s GPT family or Anthropic’s Claude), with a custom UI, a queue or workflow on top, and an audit trail. This is a wrapper. A wrapper can be a good product. The issue is paying enterprise-AI prices for one.

The wrapper test is two questions. Both answers belong in writing.

Which underlying model does your product use, and what is your contractual position with that model provider on data retention and training?

Beyond the underlying model, what specifically is your differentiated capability? Training, tuning, retrieval, eval, governance, or another concrete layer?

If the answer to the second question is “our prompt” or “our user interface,” you are buying a wrapper. The price should reflect that.

What good vendor diligence produces

A useful vendor memo is short and structured for a busy buyer. It contains, in order:

• A plain-English read of what the product actually does, with the marketing stripped.
• A clear assessment of whether it fits your use case.
• Red flags specific to this pitch and your use case.
• Five to seven sharp questions to put to the vendor in writing. Watch how fast they answer, and which ones they answer with specifics.
• Two to three alternative approaches with their trade-offs. A paid pilot, an internal build on the raw API, or staying with the current process are all credible alternatives.
• A clear Go, Maybe, or Stop recommendation with rationale.

The AI Vendor Diligence tool produces exactly this format automatically from pasted pitch material, in about a minute, for $95 AUD per pitch. You can see a sample memo as a PDF before paying.

When the stakes justify having me in the room

For six-figure annual contracts, the memo is preparation for the decision. The full version of vendor diligence is putting the questions to the vendor in writing, watching how they answer, and having an outside read in the room when you do. I do that as part of fractional work billed on my day rate.

How much diligence each contract needs

AI vendor diligence is a procurement job that many buyers have not yet set up for AI. The patterns above are reliable enough that you can run your own diligence with the checklist and a clear use case. For serious contracts, get a structured memo before the second meeting. For the largest contracts, get someone in the room when the questions are put.

The tool below produces a memo automatically. If you want to run one against a pitch you already have on your desk, paste it in; the memo is yours in about a minute.