The standard security selection runs on a requirements matrix. Several hundred rows, a weighted score, three vendors, a bake-off.
It produces a defensible decision and frequently a bad one, for a reason that is structural rather than procedural: in mature security categories, the products largely do what they claim. Detection rates converge. Feature lists converge. Everyone integrates with everyone. A matrix that scores capability will produce three numbers within a few percent of each other, at which point the decision gets made on price, on the incumbent relationship, or on whoever ran the better demo.
The variables that actually determine whether the tool works in your environment are not on the matrix, because they are not properties of the product.
What actually differs
Four things, none of which appear in a capability comparison.
Whether you can operate it.
Most security products are sold on their ceiling and lived with at their floor. A platform that requires a dedicated engineer to tune, and two more to run at the level demonstrated, is a different product in a team of six than in a team of sixty. The honest question is not what the tool can detect. It is what it will detect given the attention your team can actually give it, in the third quarter after purchase, when the person who ran the implementation has moved on.
What it does to your alert volume.
Every additional tool generates findings, and findings that nobody triages are worse than no findings — they create documented awareness of a risk you did not act on, which is a materially worse position both operationally and legally. Before adding a product, the question is who reads its output and what they stop reading to do so.
Where it sits relative to what you already own.
Security categories overlap far more than their names suggest, and the overlap grows every year as platform vendors absorb adjacent functions. A meaningful share of the capability in any new purchase is usually already licensed somewhere in your estate, unused, because nobody turned it on. That is not an argument against buying. It is an argument for establishing what you already have first, which almost nobody does, because it is unrewarding work and the vendor will not do it for you.
What it costs to leave.
Security tools accumulate configuration — tuned rules, suppression lists, integrations, historical data, and institutional habit. The exit cost is rarely estimated at purchase, and it is what turns a three-year decision into a nine-year one. Ask during evaluation what leaving looks like. The quality of the answer is informative regardless of its content.
A different sequence
The sequence below inverts the usual order deliberately. Capability appears fifth.
| Step | Question | Why it comes first |
|---|---|---|
| 1. Threat | Which specific threat are we buying down, and what is our evidence that it is our exposure? | Without this, you are buying a category because it exists. Most oversized security estates were assembled one reasonable purchase at a time. |
| 2. Inventory | What do we already own that addresses this, licensed or shipped, whether or not it is switched on? | Frequently ends the evaluation. Always changes the budget conversation. |
| 3. Operate | Who runs this on a Tuesday, and what do they stop doing? | Determines whether you are buying a product or a product plus two headcount. |
| 4. Consequence | What happens to alert volume, and who triages the increase? | Unread findings are a liability, not an asset. |
| 5. Capability | Given all of the above, which products can do the job? | This is where most evaluations start. By this point the shortlist is usually two, not seven. |
| 6. Exit | What does leaving cost in year three? | The only question vendors are never asked and always have an answer to. |
On weighted scoring
Weighted criteria are useful and routinely misused. The weights are usually set after the criteria are written, by the group that wrote them, which means they encode what the evaluators already believed.
Two disciplines make them meaningful. Set the weights before seeing any vendor, and record them. And insist that at least one criterion be capable of eliminating a vendor outright — if every criterion is tradeable against every other, the matrix cannot produce a decision, only a ranking that someone will overrule.
The most common failure is a scoring model with forty criteria at similar weights. That does not represent a considered view. It represents an unwillingness to say what matters.
The questions worth asking a vendor
Not feature questions. Those are answered in the documentation and the answer is always yes.
Ask what this product is bad at, and treat a non-answer as an answer. Ask for a reference in an organization of your size with your team size — not your industry, your staffing. Ask what proportion of their customers use the capability being demonstrated, which is a different question from whether it exists. Ask what happens to your tuned configuration if you stop paying. And ask what changed in the product after their last acquisition, or after they were acquired.
The decision that accumulates
The most consequential security decision most organizations make is not which product to buy. It is how many products to run, and that decision is never made explicitly — it accumulates.
Each purchase is individually justified. The estate that results is operated by a team that did not grow at the same rate, which means capability was bought and attention was not. A smaller number of well-operated tools outperforms a larger number of partially configured ones, and no vendor in any category will ever tell you that.
So before the evaluation starts, answer one question: what are we turning off? If the answer is nothing, the selection process is not the problem you need to solve.
This piece is a methodology rather than a product assessment. It names no vendors and makes no capability claims, and is intended to be read alongside the relevant CIOPages buyer guide, which does both.
Related Reading
- SIEM & Security Analytics
- Endpoint Detection & Response (EDR/XDR)
- Managed Detection & Response (MDR)
- Vulnerability Management Platforms
- Security Orchestration & Automation (SOAR)
- Zero Trust Architecture: From Framework to Enterprise Implementation
- Infrastructure Security in the Cloud Era: Beyond Perimeter Defense
Independent. No sponsorships. Unsubscribe anytime.