During our January Round Table about what we wanted to write about for the myriad security threats facing companies as 2021 begins, the team promoted much flashier ideas. Nonetheless, after some frank discussions, our biggest concern for our clients right now is…Fatigue.
2020 was brutal on everyone. The pandemic was a global phenomenon that impacted every living creature. The threat sequestered us all in our homes hunkered down for both work and play. The majority of non-essential workers are still operating from their homes. The lack of diversity in our daily routines tends to dull our senses. The problem compounds considerably when talking about information security practitioners. With the rise of threat actors and highly-complex, intelligent attacks, the din of incoming noise has never been louder or more threatening.
Security engineers, managers, and all the C-suite personnel with “Security,” “Audit,” “Compliance,” “Risk” and even “Technology” in their titles, are continually inundated from alerts as more security tools are added to enhance each company’s defense-in-depth strategy. Make no mistake; the added tools and the automated nature of them are a true blessing and make our lives easier. However, the problem stems from the fact that these tools all require considerable tuning and maintenance. Having worked in and around information technology for well over twenty years now, I have come to realize that technology, as evidenced by AI, really is eerily similar to a living, breathing human being. Each system and each tool used to monitor or maintain an IT system has its own manufactured parts that serve as the body and software like firmware or operating system that serves as the brain. To put it as simply as possible, as changes, even routine system patches, occurring in one part of the technology organism (i.e., stack) impacts all other interfaced (or connected) systems. Where you generally see the largest impact of infrastructure changes is with the monitoring tools for system health and security threats.
The monitoring tools are built to interface with vendors from all over the world. The more interfaces, the more value the tools have to users. Unfortunately, those interfaces and the monitoring tool’s ability to digest and understand data from a large number of devices is limited. The manufacturers and service partners for those monitoring tools simply cannot adapt as fast as the connected devices are changing. Thus, you get hiccups. And with the hiccups, you get a vast amount of noise.
Many of our security tools are programmed to alert us when they go offline. If they are offline for 1 minute, you get 1 alert. If they are offline for 60 minutes, you may get up to 60 alerts. As patches are rolled out through the technology stacks, the monitoring tools can no longer process the incoming logs from a connected system. As a result, it will throw off these error alerts.
We have dozens of different, yet similar examples we could provide, but you get the drift. Again, these tools are a boon to the security industry. Without them, we would struggle to efficiently aggregate and analyze data. With that said, they tend to come at a huge cost to those of us in the industry. That cost is noise and, yes, fatigue.
Here is what we often see: after tuning and configuring our alert settings, we begin to receive the 20-50 critical alerts per day, as we along with Management have defined them. At these levels, we find we have both the time & resources to:
- Investigate these potential threats,
- Provide recommendations and create steps to address, and
- Remediate the cause of the alert or close as false-positive.
However, each time the tools get out-of-tune, those 50 alerts can become 500 alerts or worse!
Enter FATIGUE.
The fatigue becomes most poignant because, while we now have to spend time figuring out how to filter out the noise, we are wondering what might be lurking in the midst of this noise that is signaling we have a breach.
I hope this gives the reader an appreciation for the difficult jobs your IT and Security teams have. The fatigue issue is a very real concern for us as we assist our clients with their security, audit, and compliance functions. We address these issues by discussing our resource and alert fatigue concerns with Management during the annual risk assessment, as well as during the quarterly Security Committee meetings where “workload” is always a topic of conversation.
During these meetings, we want to ensure that tickets and action items created by the Security and IT teams are addressed in a timely manner. We do this by obtaining reports of outstanding tickets, then based on severity and days outstanding, we escalate critical items to Management. Escalating issues helps the Security and Compliance teams feel supported and gives their hard work a voice.
Conversely, we also have to report to Management when our routine internal controls testing turns up less than satisfactory results with regards to control execution or evidence. It is a difficult thing to do to our brethren in Information Technology and Security, but we all understand the necessity of it. Information Security functions require continual improvement.
The visibility and transparency can lead to much-needed attention and, oftentimes, the addition of necessary resources. Bringing these issues to light ultimately results in improvement of processes, time management, and (most importantly) providing some relief from the fatigue that has been building up among the teams.
