Evaluating AI Features in Security Products
If you sat through vendor briefings this spring, you heard a lot about AI. Assistants that summarize alerts. Agents that investigate incidents on their own. Natural-language search across every log source. Automated detection tuning. Some of it is genuinely useful. Some of it is a demo that works well on the vendor’s data and much less well on yours.
AI belongs in security tools, and the better features already save real time. The challenge for security leaders is telling durable capability from presentation, before a renewal or a new purchase commits you for three years.
These are the questions we’ve found most useful.
Does it work on our data?
Demos run on curated datasets. Production environments have incomplete logs, inconsistent naming, custom applications and years of accumulated configuration.
- Ask for a proof of concept on your own telemetry, for at least a few weeks, with your own analysts using it.
- Pick test cases from your own history: a few real incidents, several common false positives and something unusual. See how the feature handles each.
- Check what happens when data is missing. Does the assistant say it doesn’t know, or does it fill the gap with a confident guess?
The last point matters most. An AI feature that’s wrong with confidence is worse than no feature, because analysts learn to trust it.
Can we see why it reached its conclusion?
Analysts need to verify AI output before acting on it, especially for containment decisions.
- Does the feature show which data it used, with links to the underlying events?
- Can an analyst reproduce its reasoning with ordinary queries?
- Are its actions logged in enough detail for an investigation or audit?
Features that present conclusions without evidence tend to be either blindly trusted or ignored. Neither is a good outcome.
What can it do without a person?
Agentic features that take actions, such as isolating hosts, disabling accounts or changing configurations, need careful scoping.
- Which actions can it take autonomously, and which require approval?
- Can you configure those boundaries per workflow and per asset type?
- What identity does it act with, and what permissions does that identity hold?
- How do you roll back an action it shouldn’t have taken?
Some organizations will want autonomous containment for high-confidence detections. Others won’t. The product should let you decide, and should make the boundaries clear.
Where does our data go?
AI features often send data to a model hosted by the vendor or a third-party provider.
- Which model providers process your data, and where?
- Is your data used to train models, including the vendor’s own?
- How long are prompts and outputs retained?
- Can sensitive data sources be excluded?
These questions belong in the security and privacy review alongside the feature evaluation, before the contract is signed.
How will it change, and who controls that?
Model updates can change behavior overnight. A feature that worked well during evaluation may behave differently after an update.
- How does the vendor test model changes before releasing them?
- Are customers notified of significant changes?
- Can you pin a version, or stage updates?
What does it cost to run?
AI features increasingly carry usage-based pricing, through tokens, credits or query limits. Model the cost at your actual volume, including growth, and ask what happens when you exceed the allowance. A feature that gets rationed because of cost won’t deliver the value it promised.
Is the feature itself secure?
An AI feature with access to your security telemetry and response actions is a privileged system. Ask how it resists prompt injection from data it processes, such as attacker-controlled strings in logs and email, and how the vendor tests for it. Log entries are attacker-influenced content, and an assistant that reads them can be steered by them.
Putting the answers together
No product will answer every question perfectly, and that’s fine. The point is to understand where a feature is strong and where you’ll need compensating controls or human review. A feature that does one task reliably, with clear evidence and sensible boundaries, is usually worth more than an agent that claims to do everything.
A structured evaluation, run against your environment and your team’s real workflows, turns a compelling demo into a decision you can defend. It also gives you a baseline for measuring whether the feature delivers what was promised after deployment.