Why Measuring What Employees Do Matters More Than Tracking What They Complete
In the realm of cybersecurity, the dynamics of employee engagement and awareness are under continuous scrutiny, particularly concerning phishing threats. A recent article by Uma Ramani highlights a prevalent routine that unfolds in organizations globally every quarter. This routine typically involves the dispatch of emails, assignments of e-learning modules, and the execution of phishing simulations that fill organizational dashboards with fluctuating defaulter counts. As these statistics rise and fall, security teams often report them to the board as indicators of organizational awareness towards potential threats.
However, this leads to a critical question: Does a reduction in defaulters signify improved employee awareness, or does it simply reflect the efficacy of the phishing simulation, with less convincing fake emails used in a given month? Many current programs lack the data necessary to accurately answer this question, posing a significant risk that the industry is beginning to confront.
Gartner distinguishes between traditional security awareness training—which focuses on tracking completion rates (who finished the training versus those who received reminders)—and security behavior and culture programs aimed at assessing whether employees change their behavior in response to actual threats. Unfortunately, most organizations remain focused on the former metrics, naively assuming they are valid indicators of the latter.
The article underscores the inconsistency of reported data, revealing how continuous training can reduce susceptibility to phishing attacks by approximately 79% over a year. Yet, this effectiveness diminishes significantly within six months without continuous reinforcement. A scatter of fluctuating success rates—where one moment a program appears effective, only to later regress—often yields no real insight beneath the noise. Furthermore, the method of addressing defaulters, typically by flagging them to their managers, can backfire. Research indicates that shame-based tactics often lead to disengagement instead of vigilance, potentially exacerbating future outcomes.
The design and implementation of training programs are just as crucial as their intent. A recent study from the CHI 2025 Conference on Human Factors in Computing Systems revealed a stark 34-percentage-point difference in effectiveness between two interventions designed for the same behavioral target. This illustrates that the method of delivering a nudge can significantly impact its efficacy. Increasingly, researchers advocate for measuring the rate of employee reporting of suspicious emails rather than simply focusing on click rates as a more reliable indicator of actual behavioral change.
Most compelling, the evidence points toward two crucial transitions in training methodologies: real-time coaching that coincides with risky behavior, rather than delayed reinforcement through scheduled modules, and the necessity for content that is tailored to individual roles and specific risk profiles, rather than adopting a one-size-fits-all approach.
One innovative idea emerging from these findings is the concept of an artificial intelligence-powered voice mechanism. Instead of re-administering tests after incidents, this system would reach out to employees following a lapse, guiding them through what transpired. Rather than following a scripted set of rules, it would facilitate a conversation that encourages employees to reflect on what made the email appear legitimate, what warning signs were overlooked, and what precautions to take next time. This method aligns perfectly with existing evidence: it is timed appropriately, personalized to individual reasoning, and significantly reduces the shame associated with mistakes.
While this approach is well theorized, it remains unproven in practice. The closest evidence base for AI-driven voice and chatbot coaching, used for promoting health behavior changes, demonstrates promise but only modest results. Approximately 82% of related studies suggest positive outcomes, but merely 36% show substantial or significant effects. Factors such as tone, trust, and execution play critical roles in determining success, yet none of these elements have been rigorously tested on the scale required for enterprise-level cybersecurity.
In conclusion, the industry is aware of the pressing questions it must confront. Moving forward, the need for large-scale, comprehensive testing to evaluate whether innovative methods of training can genuinely alter employee behavior has never been more urgent. By shifting the focus from traditional metrics of completion to a deeper understanding of behavioral change, organizations can fortify their defenses against evolving cyber threats.