AI has been on everyone’s lips for the last few years, and RSAC pushed the “AI hype” even further. Whether you’re a full believer or a major skeptic, it’s here to stay. The real use cases vary wildly — from lowering the barrier of entry for attackers, to empowering L1 analysts to do more, to more efficient SOAR workflows. Sometimes it feels like we’re bombarded with potential.
Noise, metrics & trust
When it comes to noise, we’re talking about both marketing claims and alert fatigue. New tools are added to empower your SOC team across various workflows — but how do you know what actually benefits your business? You need to identify the key metrics to track, set appropriate prioritization, and then make the right decisions when investing.
No one can hand you the single “most important” metric — yet different vendors each claim theirs is the most crucial. In our evaluation we combined our own research with public information on SOC workflows to establish key metrics. Forrester’s approach categorizes them as strategic, operational and tactical. Classic metrics like detection accuracy and false positives matter, but businesses need to establish what’s most important for them.
We track 9 metrics across those categories by splitting the evaluation into different operations, each with its own goal:
- Detection accuracy & false positive rate
- Dwell time / mean time to detect
- Alert efficiency
- Time to contain & remediate
- Time to initial understanding
- Time to advise & implement
To trust these metrics they must be looked at in the context of each other, never on their own. That gives customers a real picture of how a tested solution benefits their own needs.
How is this evaluated?
This is the only evaluation that looks at these metrics together, and it’s a complex procedure with unprecedented cooperation between teams. We designed a complete target organization — target employees, relationships with their own external vendors, normal staff workflows during the evaluation, and threat prioritization. You can read more in our methodology registry and in upcoming posts.
AI can be costly in both dollars and person-hours. The promise of more efficient workflows — better detection, lower false positives — is appetizing for decision makers. Before you commit to a new AI companion, tool or solution, consider the priorities you’ve set in your long-term strategy. Our evaluation is here to help — get in touch or encourage your vendor to take part.