What is automated penetration testing? Hint – not a pentest

In recent years, the term “automated penetration testing” has emerged in the industry as an attempt to rebrand the age-old term “vulnerability scanning,” giving a new name for either the very same activity or for a glorified vulnerability scan with a tiny bit more intelligence, leading to many believing it can replace human expertise and provide a reasonable balance of security and cost-benefit, finally making traditional penetration testing obsolete.

We are regularly approached by potential customers asking if we are able to provide automated penetration testing services, how much such services cost, which automated tools we use, and if their systems and platforms will be secure after the automated penetration test assessment.

This led to us writing this post to debunk the myth of “automated penetration testing” as a one-size-fits-all solution by highlighting the benefits it brings and underscoring its severe limitations compared to human-powered manual penetration testing.

The benefits of “automated penetration testing”

There are upsides to using automated security tools, especially for SMBs and companies with little cybersecurity maturity. Using such automated software can be a solid first step to performing regular security assessments and a good starting point before building a more robust IT security program.

“Automated penetration testing” and vulnerability scans offer advantages that can help organizations in their cybersecurity efforts, and it’s essential to recognize the areas where they excel and what purpose it is fit for.

Speed and efficiency

One of the main selling points of automated security testing tools is their ability to quickly identify security vulnerabilities across a wide range of systems. With the use of automated tools, organizations can scan their networks and applications in a fraction of the time it would take a human pentester to perform the same tasks.

While it is true that speed is an upside for automated tools, it’s important to underscore that using automation without knowing how to operate it properly can cause service unavailability and generate a lot of unnecessary noise in monitoring systems.

Scalability

As organizations grow, so do their networks and systems, making it increasingly challenging for human experts to keep up with the pace of expansion. Automated penetration testing offers a scalable solution that can be easily deployed across an organization’s expanding infrastructure, ensuring continuous security testing for known vulnerability patterns.

Cost-effectiveness

Hiring a specialized cybersecurity firm or a full-time penetration tester can be costly, especially for smaller organizations with limited budgets. An automated testing tool can be a more cost-effective option, allowing businesses to conduct regular security assessments without breaking the bank.

Continuous security testing

With the ability to run scans around the clock, “automated penetration testing” can provide continuous monitoring of an organization’s systems, identifying known vulnerabilities as they arise and alerting security teams in real-time. This proactive approach can help organizations stay ahead of potential threats and respond quickly.

The downsides of “automated penetration testing”

Despite the advantages outlined above, relying solely on automation for security assessments can leave organizations exposed to various risks and prone to be successfully attacked by threat actors with minimum levels of sophistication. Understanding the downsides and limitations is crucial for making informed decisions about your cybersecurity strategy, when to use automation and when to prioritize a different approach.

Limited in nature: only covers known vulnerabilities

Despite what marketing departments from vendors that produce security testing tools might say, the current state of “automated penetration testing” software relies on predefined vulnerability databases, version banners of target software known to be prone to vulnerabilities, attack signatures and publicly known techniques such as common misconfiguration, password spraying and lists of default passwords to identify potential weaknesses. This shortcoming means that automated scanners might not discover security weaknesses that have not been publicly disclosed or do not follow a repeatable pattern.

As a result, scanners can only detect low-hanging fruit issues or issues for which they have signatures, making them ill-equipped to uncover novel vulnerabilities or specific and unique risks to an organization – unlike experienced pentesters who may identify attack paths through creative, out-of-the-box thinking, taking into account the full context of the assessment, the target organization, to discover previously unexplored pathways into your corporate secrets.

False positives and false negatives

The very nature of a vulnerability scan or “automated pentest” can lead to both false positives (identifying vulnerabilities that don’t actually exist) and false negatives (failing to detect real vulnerabilities).

It’s no secret that vulnerability scanning tools have significantly improved in recent years, but they are still known for false positive rates that are far from ideal, usually alerting on security issues based on a banner version without proof of exploitability, prioritization metrics or gauging the real risk it presents to the environment. These inaccuracies can waste valuable resources and time, as security teams must manually verify the results and put the bigger picture in context before allocating resources to address the identified vulnerabilities.

Lack of context and business logic understanding

Virtually every “automated penetration testing” tool lacks the ability to understand the context, the business logic of an organization’s applications and the intricacies of trust relationships between interconnected systems in any sufficiently complex network.

This limitation can result in the tools missing to identify non-obvious vulnerabilities that a human penetration tester, with their understanding of the organization’s processes and business, could easily spot.

Limited ability to perform multi-step attacks

Some cyber threats involve a series of steps that individually may not appear malicious but, when combined, lead to a successful attack chain. Most such tools often struggle to identify multi-step attack vectors, even those with a small degree of complexity, due to their reliance on predefined patterns and signatures.

As an example from a recent engagement we performed, we used automation to kick off the initial reconnaissance phase against the customer’s infrastructure, and as a result, we discovered a directory listing where a handful of JavaScript files were on display.

Secondly, we downloaded the JS files and used some more automation to inspect them for high-entropy data to look for API keys and other secrets that could perhaps be stored in cleartext in those files. Lo and behold, we found some secrets related to debugging functionalities in a certain PHP framework.

Then as a third step, our team had to manually figure out how to use the debugging key we had just found. This meant several hours of research on how to serialize PHP objects and tweak the output in a way that allowed the application to successfully deserialize the objects, which the team used to get an object injection primitive that was finally escalated into a remote code execution and granted our team initial access and unfettered privileges in the server. Consequently, it served as the base for further attacks to compromise other assets within the client’s network.

Albeit a relatively uncomplex attack chain, this is not something the majority of “automated pentesting” tools can come close to achieving in their current form.

It is true, however, that there is an emerging generation of tools with machine learning and AI capabilities that will change a lot of the landscape of the current tools, and we are yet to see how effective they can be at replicating more elaborate pentesting tasks. Currently, many of these tools have better performance at replicating tasks often encountered in internal penetration testing, but again they follow the same model of a vulnerability database and known security issues, not yet able to find new vulnerabilities by themselves.

Inability to simulate social engineering attacks

Most high-profile hacks start with phishing and other social engineering tricks as the main vector for initial access. “Automated penetration testing” software is, for the most part, unable to simulate convincing social engineering attacks, which are tactics that manipulate individuals into revealing credentials and other sensitive information.

An experienced pentesting team, on the other hand, can employ social engineering techniques, such as phishing, SMishing and pretexting, or perform authorized business email compromise attacks, to further escalate access into an organization.

Noisy attacks and difficulty in carrying out stealth pentests

Scanners are notably noisy because they generate a lot of traffic in a very short period when attempting to discover vulnerabilities. For example, an automated vulnerability scanner might fire off thousands of requests in a minute, a minimum of one for each attack signature it might have in its vulnerabilities database, in a continuous fashion, leading to a high amount of unwanted traffic and bandwidth usage.

This behavior is responsible for triggering alerts in monitoring teams and sometimes even causing a denial of service situation, making “automated pen testing” tools unsuitable for security testing where discretion is needed.

Real-life cyber attacks happen differently, beyond what automation can offer

Real-world cyber intrusions go far beyond merely automated vulnerability scanning. It’s true that state-sponsored APT groups and professional red teams that perform adversary simulations do have a lot of custom tooling and classified techniques, but in real-life incident response reports, one can rarely see mentions of automated scanners being used in the majority of the processes in the cyber kill chain.

As an example of a recent report by DFIR Report dated April 3rd, 2023, one can see that even the initial access vector is complex enough to make it difficult to be replicated purely by automation in a way that it is stealth enough not to be detected:

Below is an infographic listing the advantages and disadvantages of “automated penetration testing”:

The value of human ethical hackers and manual penetration testing

While “automated penetration testing” and vulnerability scanning tools offer tangible benefits, they are still far from being suitable to replace the expertise and adaptability of human ethical hackers. Here are some key reasons why human expertise is essential in cybersecurity assessments:

Critical thinking and creativity

Experienced penetration testers possess the ability to think critically and creatively, allowing them to identify vulnerabilities that automated tools might miss. By considering the unique context of each system and application, they can uncover novel attack vectors and custom threats that a tool may not be programmed to detect.

Adaptability and learning from experience

As cybersecurity threats evolve, so must the methods used to counter them. Human ethical hackers can learn from their experiences, adapt their techniques, and apply their knowledge to new and emerging threats. Automated penetration testing tools, on the other hand, are limited by their pre-programmed rules and vulnerability databases.

Understanding the context and business logic

Seasoned pentesters can grasp the business logic and context of an organization’s systems and applications, allowing them to pinpoint vulnerabilities that automated tools might overlook. By considering how an organization operates, they can target weaknesses that may have serious implications for business continuity and data protection.

Ability to execute targeted, tailored attacks

In some cases, organizations may require targeted, tailored attacks to assess specific aspects of their security posture. Human hackers can customize their approach to meet these unique needs, while “automated pentesting” tools may struggle to adapt their predefined scans to suit each situation.

The ideal approach: combining automation with human expertise

To ensure a robust and comprehensive cybersecurity strategy, organizations should consider combining automated penetration testing with human ethical hacking.

Using automated tools for initial vulnerability scanning

Automated security tools can and must be employed for an initial vulnerability assessment, quickly identifying known weaknesses across a wide range of systems. This initial scan can help organizations prioritize their security efforts and allocate resources efficiently.

In our penetration testing engagements, automation plays an important role. Out of the entirety of our engagements, it’s safe to assume that automated tools are almost always used, and around 20% to 25% of the engagement relies on them to find common security misconfiguration and known vulnerabilities because scanners excel at it and are able to perform such tasks at scale. The output from such tools is invaluable in serving as a guide to continue the assessment, as they frequently bring up good leads and pointers to chase.

Human pentesters performing manual testing and verifying results

Following the automated scan, a security tester can perform manual testing to validate the results, explore potential false positives and negatives, and uncover vulnerabilities and attack paths that automated tools overlook. By doing so, they can provide a more thorough and accurate assessment of an organization’s security posture and the cyber risks it is subject to.

Continuously updating and refining automated tools based on human-powered findings

As ethical hackers uncover new vulnerabilities and attack vectors, their findings can be used to update and refine automated penetration testing tools. This iterative approach can help organizations stay ahead of emerging threats and ensure their automated tools remain effective and relevant.

Conclusion

While “automated penetration testing” and vulnerability scanners offer advantages in terms of speed, scalability, and cost-effectiveness, they are currently unsuitable to replace the expertise and adaptability of human ethical hackers performing manual penetration testing.

By understanding the limitations of automated penetration testing software and recognizing the value of human expertise, organizations can develop a comprehensive security strategy that leverages the best of both worlds in their penetration tests.

Combining automated vulnerability scanning with manual testing and continuous refinement ensures a robust defense against ever-evolving cybersecurity threats.

FAQ