Buying AI: vendors and models
Every vendor now sells AI, and the contract says little about what happens to your data.
Ask the questions that decide data use, model changes and exit before signing, test candidate models on your own tasks, and price the result per task. Start with the questionnaire; it shortens the list.
Downloads
Files you keep and edit: PDF, Excel, Word or PowerPoint. Most show part of the file before you ask for it.
- Questionnaire and scorecardAI Vendor Due Diligence Questionnaire38 questions for an AI product vendor in ten groups, each with a good answer and a red flag, scored into a weighted total.
- ScorecardLLM Evaluation ScorecardA scorecard that tests up to four LLMs on your own test set: accuracy, groundedness, refusals, format, latency and cost per task.
- ModelAI Unit Economics ModelAn Excel model of the cost of one completed task for an AI feature, against today's cost, with break-even volume and sensitivity to price and retries.
Online tools
Fill in the page and get the result on screen. No account needed.
Apps
Enter your situation and get the analysis back: a score, a draft or a plan. They run with a CIOPages account.
Part of the job Choose and buy technology.
Other sets
- Management consulting tools for CIOs
- Deciding where AI goes
- Cutting cloud and SaaS spend
- Governing the AI already in use
- Buying technology without regret
- AI regulation and risk
- Architecture decisions that hold up
- Technical debt you can show the CFO
- Security platform evaluations
- Running IT like an investment portfolio
- Org design and decision rights
- The first 100 days as a CIO
- Board and budget season
- Incidents and resilience
Keep reading
A scorecard that tests up to four LLMs on your own test set: accuracy, groundedness, refusals, format, latency and cost per task.
A script for vendor reference calls: finding customers the vendor did not pick, 20 questions in five groups, red flags, and a scoring sheet.