AI vendor pitches have never been more polished. Slick decks, impressive demos, case studies from recognizable brands, and a sales team that has a confident answer to every question you ask. That confidence is the first thing you should be skeptical of.
Buying AI is not like buying software. Traditional software either does what it says or it does not. AI systems operate on probabilities, training data, and model assumptions that your vendor controls and you cannot see. When something goes wrong — and eventually something will — you need to know what you actually bought, who owns the problem, and what your options are.
Most executives find out the answers to those questions after signing. Here is how to find them before.
The Demo Is Not the Product
Every AI demo is run on curated inputs in a controlled environment. The vendor has selected the examples that make the system look best. That is not dishonest — it is how demos work. But it means the demo tells you almost nothing about how the system will perform on your data, in your environment, on the edge cases that actually matter to your organization.
Before you evaluate any AI vendor seriously, ask to run the system on your own data. Not sample data, not similar data — your actual data, including the messy cases and exceptions that exist in every real organization. If the vendor resists this, that is your answer. A system that only performs well on clean, prepared inputs is not ready for production.
Also ask specifically how the system handles cases it is uncertain about. Does it tell you when it does not know? Does it escalate, or does it produce a confident-looking answer anyway? Systems that fail silently are far more dangerous than systems that fail loudly.
Ten Questions That Separate Real from Theater
These are the questions that matter. Not every vendor will have good answers. The quality of the answer tells you as much as the answer itself.
1. How was the model trained, and on what data? You need to understand what assumptions are baked into the model. Data from a different industry, a different regulatory environment, or a different customer profile does not automatically transfer. Push past “large dataset” to specifics.
2. What is your accuracy on tasks like mine? Ask for performance benchmarks on tasks that match your use case. Ask how those benchmarks were measured, who measured them, and whether you can verify them independently. Vendor-reported accuracy on vendor-selected benchmarks is not useful.
3. How does the system perform when conditions change? Models degrade when the real world diverges from training data. What is the vendor’s process for detecting and responding to drift? How will you know when performance has degraded?
4. Who is accountable when the output is wrong? This question makes vendors uncomfortable, which is exactly why you should ask it. When the AI makes an error that causes a problem for your organization, what is the vendor’s liability? What does the contract say? Get a clear answer and make sure the contract reflects it.
5. What data do you retain, and how is it used? Does the vendor use your operational data to train future models? Who owns the outputs the system produces? Where is data stored and under what regulatory framework? These are not IT questions. They are governance questions with legal and competitive implications.
6. What happens if you change the model? Vendors update models. Sometimes dramatically. What notice will you receive before a model update changes system behavior? What is the process for testing changes before they affect your operations? You need contractual answers, not assurances.
7. Can I audit the system’s decisions? For any AI system involved in consequential decisions, you need an audit trail. Can you review the inputs and outputs for any given decision? Can you explain to a regulator or a customer why the system produced a particular result? If the answer is no, that is a governance problem before it is a technology problem.
8. What does migration look like? You will eventually change vendors, retire the system, or bring capability in-house. Can you export your data in a portable format? What is the offboarding process? Vendors who make it difficult to leave are pricing the exit into your relationship from day one.
9. What are your uptime and SLA commitments? If your operations depend on this system, what guarantees do you have about availability? What is the remediation when SLAs are missed? “We work hard to maintain uptime” is not an SLA.
10. What does your customer list look like six months after signing? Ask for references from customers who have been live for at least six months, not pilot customers. Ask those customers specifically about what did not go as planned. Every implementation has friction. The question is whether the vendor helped navigate it or disappeared after the sale.
Red Flags That Should Stop the Conversation
Some vendor behaviors are not caution flags. They are stop signs.
Any vendor who cannot clearly explain how the model works in plain language is either hiding something or does not know themselves. Neither is acceptable for a system you will depend on operationally.
Any vendor who tells you governance and compliance are “something we can figure out together after you’re live” is asking you to assume the liability they are not willing to accept. Walk away.
Any vendor whose pricing structure makes it expensive to leave — through data lock-in, proprietary formats, or exit fees — is telling you how they view the relationship. You are a captive, not a customer.
Any vendor who cannot provide actual customer references, only case study documents, should be treated with significant skepticism.
Governance Before Contract
Before you sign anything, answer three internal questions that are entirely within your control.
First: who in your organization will own this system post-deployment? Not the IT team that implemented it, not the vendor. A named executive who is accountable for what the system does. If you cannot answer this, you are not ready to sign.
Second: what is your monitoring plan? How will you know the system is performing as intended three months from now, six months from now, after the vendor updates the model? Monitoring needs to be designed before deployment, not after an incident.
Third: what is your exit threshold? Define in advance the conditions under which you will terminate the relationship. Performance below a certain level, a security incident, a pricing change beyond a certain percentage. Having a clear exit threshold protects your negotiating position and forces clarity about what success actually means.
Speed and Rigor Are Not Opposites
The pressure to move fast on AI is real. But due diligence on an AI vendor is not about slowing down. It is about making sure that when you commit organizational resources and operational dependency to a vendor, you are doing it with clear eyes.
Organizations that skip this process do not save time. They spend the time they saved on remediation, legal exposure, and rebuilding trust after something goes wrong.
If you are not sure where your organization stands before entering vendor conversations, our AI Readiness Assessment gives you an honest baseline in under ten minutes. If you want a deeper framework for governance before deployment, the Executive’s Guide to AI Readiness covers the full picture. Both are free, and both will sharpen the questions you bring to any vendor conversation.
The vendors worth working with will welcome the rigor. The ones who do not have told you everything you need to know.