Back-of-the-cocktail-napkin math

February 15, 2018

Our team recently attended Gartner’s Identity and Access Management conference in Las Vegas, Nevada, and we had the pleasure of meeting many first-time attendees who had not yet deployed a Privileged Access Management (PAM) program. This might come as a surprise, as there was a landslide of panic-driven purchasing in response to regulatory controls such as PCI, HIPAA, Sarbanes-Oxley, and other security directives. However, we met an astonishing number of organizations still using manual processes and open source software rather than choosing to purchase and deploy a commercial solution. However, not purchasing PAM also has its costs.

The attendees we met who did not have a PAM program all expressed the same underlying frustration that while they understood PAM technology was important, they could not get budgetary approvals. These organizations had no automated way to rotate passwords on a regular, scheduled basis. They were also generally afraid of rebooting systems, despite the agreed-upon values of clearing stored password hashes that can be obtained by tools like Mimikatz, which can scrape memory in Windows to obtain passwords and hashes.

As a result, it’s a good day to be a criminal. Databases are the highest value for insider attacks, based on Crowd Research Partners’ 2017 Insider Threat Report. And as Cyberreason found in their 2017 Threat Hunting Report, attackers dwell on a network for an average of 40 days before they’re discovered. With over a month to work, criminals can take a measured, leisurely approach when exfiltrating gigabytes of data from on-premises or cloud-hosted databases where passwords are being manually rotated on a slow schedule. That stolen data can be resold on the dark web, particularly if it’s personal identity information (PII), higher-value personal health information (PHI), or proprietary trade secrets stored in a database. The lack of constantly rotating passwords gives criminals the lag time to easily extract valuable data.

Many of the attendees we met used KeePass, an open-source tool intended for personal use but unfortunately re-purposed for storing companies’ database and local administrator passwords. The most obvious problem is that it’s nearly impossible with KeePass and similar programs to determine who checked out a password, or what they did with that password. It’s similarly problematic to discourage someone from taking an offsite text export of the main password vault. A personal free tool used in a commercial environment creates challenges by the very deficits in its design, but these problems pale in comparison to the costs associated with a free password vault.

Let’s take a look at the real costs of manual password changes. For purposes of comparing a commercial PAM solution to a ‘free’ solution, assume that a systems administrator and a database administrator are required to participate in password changes for database accounts. These two high-value personnel are required so that they can both change the password and restart the underlying database server or client applications using the embedded password. According to Glassdoor, a database administrator earns an average salary of $89,626 per year, and a systems administrator earns an average of $71,144 per year. Assume a loaded staff cost of 20% to cover paid time off, sick leave, medical and dental care, and other expenses. This results in a cost of $90.48 per hour to change database passwords.

As a database or client application restart is required, assume that the systems administrator and database administrator can change one password every twelve minutes, or five passwords an hour. This means that an organization with 100 database passwords that need to be changed every thirty days will spend $21,715.20 changing database passwords annually, and consume 240 hours of staff time per person.

Additionally, assume that the systems administrator must also manually change the local administrator password on Windows systems every thirty days. Assume this process is faster, and so they can change ten local administrator passwords per hour. With 500 systems, this means a systems administrator will spend 50 hours per month (or 600 hours per year) manually changing passwords, at an annual cost to the organization of $23,256. It is use of time that could most certainly be more efficiently directed elsewhere.

This scenario means that the systems administrator is spending 70 hours a month in total just changing passwords manually, which leads to low job satisfaction and burnout. This, in turn, can lead to needing to hire new systems administrators, which requires additional password rotations once the disgruntled system administrator leaves. The estimated costs do not include opportunity costs associated with having high-cost resources performing manual work that should be automated, or the associated HR costs of recruiting new employees. In larger organizations with more than 200 database passwords and more than 1,000 local administrator passwords, 1.5 full-time employees (FTEs) would be required to rotate passwords manually. In plain terms, it blatantly absurd not to automate this task considering the hours required otherwise.

The reality of organizations that do not automate password rotation is that they choose to consciously degrade their operational security rather than have systems administrators and database administrators spend half (or more) of their working day changing passwords. Instead, it’s very common to see the same database or local administrator password used across all the production machines, and often it’s the same password as in development. This operational reality runs counter to all known best practices to limit lateral movement of criminals breaching a network. Having the same password everywhere makes it easier for the systems administrators but at the cost of enabling attackers to have free run of a target organization.

Nearly all vendors in the PAM space price their products to be less expensive than FTEs, with the added advantage of providing technical support and continuous coverage in case of staff holiday. These solutions are also cheaper than the cost of a breach, which ranges from the mid-six figures to the upper-seven figures for most organizations, even when including the costs of consulting and three years of support. If your organization has yet to deploy a mature PAM program, we recommend setting your 2018/2019 budget to include building one to reduce the hidden operational costs of ‘free’ software. The likelihood of breaches may sound like another far-off statistic, but the consequences can be very real and personal. It simply isn’t worth the potentially significant negative business consequences to delay implementing a PAM program in the long term for most businesses.