CIOPages
All RFP packages

RFP Package · DevOps & Platform Engineering

CI/CD Pipeline Platforms RFP questions and template

128 questions, 10 demo scenarios and a five-vendor scorecard for choosing CI/CD Pipeline Platforms software, in one Excel workbook.

Buy the workbook, from $499Read sample questions

What this package is for

Use it to run a CI/CD Pipeline Platforms software selection, from the first long list to the final scorecard.

What the category covers. Questions for buyers choosing a CI/CD platform: pipelines as code, runners, source control and review gates, tests, pipeline secrets and cloud credentials, security scanning, artifacts and provenance, deployment and release, governance and audit evidence, delivery metrics, migration from a current CI tool and third-party extensions. Bought by engineering, platform and security leaders consolidating build and release tooling or standardizing how many teams ship software.

A selection usually runs in three rounds. The package has questions for each:

  • RFI, to the long list. 29 questions screen out products that lack something you need.
  • RFP, to the shortlist. 65 questions ask how each product does the work.
  • Deep dive, to the finalists. 34 questions ask for proof on your own data.

10 demo scenarios tell each vendor what to load and what to show, so every product does the same work in front of you. 110 due-diligence questions cover security, integration, implementation and exit. The scorecard weights the answers and ranks up to five vendors.

Each question comes with why it matters, what a good answer looks like and the red flags, so the people scoring the replies know what to look for.

3 questions from the package

From the RFI round. The first shows part of the guide each question carries; the workbook adds follow-ups, how to verify the answer, a priority and a weight.

1. Which pipeline settings, if any, can be set only in the user interface or API rather than in the pipeline file stored in the repository?

Why it matters. Settings held outside the repository are not versioned or reviewed with the code. A pipeline's behavior can then change without a merge request, and an older commit cannot be rebuilt the way it originally ran.

Good answer
  • Provides a written list of pipeline settings that live outside the pipeline file, such as trigger rules, variables or runner selection, with the reason each one is held there
  • Running a pipeline on an older commit uses the pipeline file committed with that commit
  • Settings held outside the file can also be managed from a configuration file kept in a repository, for example through an infrastructure-as-code tool
Red flags
  • States that everything is in code but cannot produce the list of settings held outside the file
  • Job steps or trigger rules can be edited in the user interface with no corresponding change in the repository
  • Older commits run with the current pipeline definition instead of the one committed with them

2. Provide a matrix of the operating systems, CPU architectures and GPU options available on your vendor-hosted runners and supported by your self-hosted runner agent.

Why it matters. If an operating system or architecture the buyer builds for is missing from hosted runners, the buyer must buy, run and patch its own machines for those builds. If it is missing from the self-hosted agent, those builds cannot run inside the buyer's network at all.

3. Does the product host Git repositories itself, build from repositories hosted on an external Git platform such as [current Git hosting], or both?

Why it matters. If the product cannot build from where the buyer's code lives, the buyer must move repositories or run a second tool. Which hosting option is used also determines which review, protection and status features work.

Capability areas

Pipeline-as-code authoring and reuse (13)

Covers pipeline definitions kept in the repository, multi-stage and DAG structure, conditional and manual jobs, matrix builds, triggers (push, schedule, API, upstream pipeline), monorepo path filtering, cross-repository triggers, versioned shared templates, service scaffolding from approved templates, and how pipeline syntax changes are versioned and deprecated. Centrally enforced required steps are covered under governance.

Runners, execution and build performance (13)

Covers vendor-hosted, self-hosted and hybrid runners, operating system and CPU architecture coverage, autoscaling, job isolation and teardown, outbound-only runner connectivity, access to private networks, queue behavior at peak concurrency, dependency and build caching, retry rules, and running builds during a hosted-service outage. Hosting regions and tenancy of the platform itself belong to the deployment-hosting module.

Source control and code review integration (9)

Covers built-in Git hosting where offered or integration with external Git platforms, commit and merge request status reporting, branch protection with required checks, code owners, signed-commit verification, merge queues, large-file storage, mirroring and read replicas, and repository import with history and review data. Generic identity integration is out of scope.

Test automation and quality gates (9)

Covers test result reporting per test, failure display on the merge request, code coverage and coverage change, static analysis findings on the merge request, quality gates that block merges, and temporary review environments per merge request. Security scanning is covered in its own area.

Pipeline credentials and secrets (11)

Covers OpenID Connect federation to cloud providers in place of stored keys, the claims available for cloud trust policies, encrypted secret storage and configuration variables scoped by project, environment and branch, run-time retrieval from external secrets managers, log masking, buyer-managed encryption keys for secrets, the default scope of the pipeline job token, policy limits on personal access token and deploy-key scope and lifetime, and how pipelines triggered from forks or untrusted contributors are kept away from secrets. The vendor's own corporate security program belongs to the security module.

Security scanning in the pipeline (9)

Covers push-time secret detection and blocking, dependency and container image vulnerability scanning, license reporting against a deny list, dynamic testing of review environments, display of new findings on the merge request, finding triage, scanner database update frequency and sources, and scanners that run inside the buyer's network without internet access. Signing and provenance are covered under artifacts.

Artifacts, registries and build provenance (11)

Covers container image builds, built-in or connected container and package registries, retention rules that protect release artifacts, promotion of one built artifact across environments, SBOM generation per build, artifact and image signing, build provenance attestations, and verification of signatures and attestations before deployment. The vendor's own software supply chain belongs to the third-party module.

Deployment and release orchestration (14)

Covers deployment to the buyer's targets (Kubernetes, virtual machines, serverless), GitOps controller integration, approved single-action and automatic production releases, environment approvals, freeze windows, progressive rollouts, rollback, multi-service release ordering, feature flags, infrastructure-as-code plan and apply in pipelines, release records, per-environment deployment history, and behavior when the service fails mid-deployment.

Governance, policy and audit evidence (12)

Covers policy as code across projects, central required pipeline steps teams cannot remove, role-based access down to the environment level, separation of duties for merge and deployment, immutable audit logs with retention and SIEM streaming, retention of production deployment records for archived projects, timestamp integrity, change-record integration with IT service management tools, and release evidence export. Third-party certifications belong to the compliance module.

Pipeline observability, metrics and notifications (10)

Covers cross-project pipeline and deployment dashboards, delivery metrics (deployment frequency, lead time for changes, change failure rate, failed deployment recovery time), pipeline duration, queue time, failure and flakiness trends, job log search and retention, compute minutes and storage metering with caps, notifications, chat-based actions, monitoring-tool event links, and alert-triggered pipelines. Pricing terms belong to the commercial module.

Migration and pipeline conversion (9)

Covers conversion of pipeline definitions from the buyer's current CI tool, reporting of what could not be converted, running old and new pipelines in parallel by cohort, migration of runners, secrets and variables, and the written migration plan with duration and exclusions. Data export on exit belongs to the migration-exit module.

Ecosystem and extension security (8)

Covers the marketplace or plugin model for pipeline steps, pinning third-party steps to a fixed version or digest, administrator approval or blocking of third-party extensions, the permissions an extension receives, linking commits, merge requests and deployments to work items, shipped integrations with issue trackers and IDEs, webhooks for pipeline events, and the command-line tool for pipelines and deployments. The pipeline job token, personal access tokens and deploy keys belong to CRD; generic API coverage and rate limits belong to the integration module.

Demo scenarios

Each scenario lists the data to load before the demo, then the steps to show, and the questions it scores.

  1. Converting one of our existing pipelines
  2. Deploying to our cloud with no stored keys
  3. Signing an image and blocking unsigned deployments
  4. Reviewing a risky merge request before merge
  5. Keeping a required scan stage in every pipeline
  6. A failed canary and a freeze window override
  7. A burst of jobs across hosted and self-hosted runners
  8. One service change in a shared monorepo
  9. Evidence for one production release
  10. Adding a third-party pipeline extension

Due diligence

The workbook carries the screening questions from these modules. Each module is also sold on its own.

Questions about this package

How many CI/CD Pipeline Platforms RFP questions are there?

128 solution questions in 12 capability areas: 29 for the RFI, 65 for the RFP and 34 deep-dive questions for the finalists. The workbook adds 110 due-diligence questions on security, integration, implementation and exit.

What comes with each question?

Why it matters, good-answer signals, red flags, follow-up questions, how to verify the answer (a demo step, a test or a document), and a suggested priority and weight for scoring.

Can I edit the questions?

Yes. The workbook is an ordinary Excel file. Change, add or remove questions, and change the weights; the scorecard recalculates.

Which license do I need?

The Enterprise License covers any number of evaluations inside one organization. The Consultancy License covers use with any number of clients. Neither allows reselling or republishing the questions.

Before you shortlist

The buyer guide compares the products in this category and what decides between them.

Buyer Guide
CI/CD Pipeline Platforms

For the business side of the same change: