3 questions from the package
From the RFI round. The first shows part of the guide each question carries; the workbook adds follow-ups, how to verify the answer, a priority and a weight.
1. Provide a list of your pre-built connectors for CRM, ITSM, order management and billing systems, marking for each connector which operations write data back to the system rather than only read it.
Why it matters. A connector limited to lookups lets the assistant report status but not complete the request. The customer still needs an agent, and containment stays low.
Good answer
- Each connector lists named operations, such as create case, update order or issue credit, not just a system name
- Write operations are clearly separated from read-only operations
- Supported editions or API versions of each target system are stated
Red flags
- A logo wall or system names with no operation list
- "Any system via API" offered in place of the requested list
- Write operations described as roadmap items or as delivered by professional services only
2. Provide a list of the knowledge source types and file formats your product can ingest for generated answers, marking for each whether content syncs automatically or requires manual upload.
Why it matters. If [knowledge source] cannot be ingested, or can only be loaded by manual upload, answers drift from the buyer's source of truth. Staff then spend time re-uploading content.
3. Does your product classify each customer message against intents that an administrator defines, returning the matched intent with a confidence score?
Why it matters. Without classification against defined intents, the buyer cannot make sure a transactional or regulated request reaches its fixed flow. Routing would then depend on free-form model generation.
Capability areas
Backend Integration & Action Orchestration (14)
Covers pre-built and custom connectors to CRM, ITSM, order and billing systems, function and tool calling, webhook or serverless fulfillment, write-back, multi-step transaction completion, and error and timeout handling inside a conversation. Generic public APIs, SDKs and SSO/SCIM are out of scope because the integration module covers them.
Knowledge Grounding & Generative Answers (12)
Covers retrieval-augmented answers over our knowledge sources: content ingestion and sync, chunking and indexing controls, source restriction, citations, answer behavior when no source supports a response, and content freshness. The vendor's general model evaluation methodology is out of scope because the AI performance module covers it.
Language Understanding (10)
Covers intent and entity recognition where determinism is still needed, handling of misspellings and colloquial input, out-of-scope detection, disambiguation, and carrying context across multiple turns and topic switches. Also covers understanding and answering in each language the buyer serves, detecting the customer's language and switching language within a conversation. Administration-interface languages are out of scope.
Dialog Management & Hybrid Flow Control (11)
Covers dialog state, slot filling, and combining deterministic flows with LLM generation in one assistant. It also covers how the platform keeps regulated or transactional steps on a fixed path, and digressions with return to the interrupted flow. Agent-level action permissions are out of scope because the agentic autonomy module covers them.
Conversational Guardrails & Response Fencing (12)
Covers guardrails the buyer can configure in the assistant: topic and response fencing, blocked-content rules, PII detection and redaction in live turns and transcripts, response tone controls, per-assistant prompt-injection and jailbreak settings that can be tested, and a per-turn log of which guardrail fired and what the assistant did. The vendor's platform-level red-teaming and disclosure posture is out of scope because the AI safety module covers it.
Customer Authentication & Session Context (8)
Covers identifying and verifying the end customer within a conversation: login handoff, step-up verification before sensitive actions, voice and chat ID&V options, and passing authenticated identity to backend calls and personalization. Workforce SSO for platform administrators is out of scope.
Voice & Telephony (11)
Covers voice and IVR deployments: speech recognition and synthesis options, barge-in, DTMF input, turn-taking and silence handling, latency under load, SIP and telephony connectivity, and voice-specific tuning such as custom vocabulary. Digital messaging channels are out of scope.
Digital & Messaging Channels (10)
Covers the web widget, mobile in-app SDKs, SMS, RCS, third-party messaging apps and social messaging. It also covers rendering rich elements per channel, handling channel-specific limits, and keeping conversation context when a customer moves between channels. Accessibility conformance of the widget is out of scope because the accessibility module covers it.
Human Handoff & Agent Desk Integration (10)
Covers escalation to a live agent: escalation triggers, routing data passed to the CCaaS or agent desk, transfer of transcript, intent and authenticated context, queue and after-hours behavior, and return of the customer to automation after agent handling. General human-in-the-loop review of AI decisions is out of scope because the human oversight module covers it.
Authoring, Testing & Release Lifecycle (12)
Covers low-code authoring with pro-code extension, prompt and flow versioning, separate development, test and production environments, conversation simulation, assistant-specific regression test sets, controlled promotion and rollback of releases, and importing intents, training phrases and flows from an existing bot. Vendor training programs are out of scope.
Containment & Outcome Analytics (10)
Covers how the platform measures resolution and containment as distinct from session counts. It also covers escalation-reason analysis, CSAT and effort capture, cost-per-resolution reporting, and transcript mining that surfaces failed containments and new automation candidates. Token and compute cost accounting is out of scope because the AI cost module covers it.
Model Selection & LLM Configuration (8)
Covers which LLMs the assistant can use, bring-your-own-model options, assigning different models to different tasks such as retrieval, generation and classification, fallback when a model provider is unavailable, and buyer control over prompts and generation parameters. The vendor's model change discipline and training-data provenance are out of scope because the model governance module covers them.
Questions about this package
How many Conversational AI & Chatbot Platforms RFP questions are there?
128 solution questions in 12 capability areas: 27 for the RFI, 66 for the RFP and 35 deep-dive questions for the finalists. The workbook adds 110 due-diligence questions on security, integration, implementation and exit.
What comes with each question?
Why it matters, good-answer signals, red flags, follow-up questions, how to verify the answer (a demo step, a test or a document), and a suggested priority and weight for scoring.
Can I edit the questions?
Yes. The workbook is an ordinary Excel file. Change, add or remove questions, and change the weights; the scorecard recalculates.
Which license do I need?
The Enterprise License covers any number of evaluations inside one organization. The Consultancy License covers use with any number of clients. Neither allows reselling or republishing the questions.
