The so-called “gig economy” has increasingly gained popularity throughout the past decade. Many industries, especially the services sector, have been disrupted by gig workers and the “Uber” model, relying on a network of individuals willing to perform a short-term service in exchange for a fee paid by a platform.
As companies accelerate their digital transformation and increase investments in cyber resilience, the crowdsourced model has also appeared in the cybersecurity industry, initially in the form of bug bounty programs and more recently as part of crowdsourced penetration testing through pentest as a service (PTaaS) platforms.
Crowdsourced pen testing, marketing departments of vendors in this field say, leverages the skill sets of a global community of freelance security testers. While this approach can provide benefits such as cost-effectiveness and fast turnaround to start an assessment, it introduces a number of significant challenges and risks that can jeopardize an organization’s data privacy and security testing quality, as well as other problems that are conveniently not mentioned by marketing collaterals and salespeople until you learn the hard way after signing the contract.
This article draws from our experience working with companies that also engaged in crowdsourced pentests in the past, and many of the words written in this article echo the opinions of cybersecurity leaders in mid-market and enterprise organizations that once opted to use “the crowd” for their pentest program.
We intend to demystify the hype around crowdsourced security testing, critically analyze its benefits and drawbacks, and contrast it against “traditional” penetration testing models. The post also touches on the differences between crowdsourced pentesting and bug bounty, which are often mistakenly conflated.
What is crowdsourced penetration testing?
Crowdsourced penetration testing is the “Uber model” applied to cybersecurity testing that employs the collective expertise of multiple freelance hackers, also known as the “crowd”, to evaluate the security robustness of a system, network, or application.
In the following section, we will delve into a detailed comparison of the advantages and disadvantages of the crowdsourced pentest model vis-a-vis traditional penetration testing models.
Crowdsourced penetration testing vs. traditional penetration testing
Crowdsourced penetration testing and traditional penetration testing are two distinct approaches to assessing an organization’s security posture through security testing.
Roughly speaking, the main distinction between these two approaches is the fact crowdsourced testing outsources the testing process to a pool of freelance security researchers across the globe and the delivery model via a SaaS pentest as a service, web-based platform.
Although appealing due to its simplicity and perceived advantages, the freelancer/outsourcing model comes with numerous problems that are often overlooked or not talked about. In this section we intend to discuss the differences, benefits and shortcomings of both approaches.
Versatility in testing scopes and targets
In crowdsourced pen testing, with rare exceptions, the crowd doesn’t have any useful skills that go beyond pen testing of external-facing assets related to web applications and APIs and, with some luck, mobile apps. Other types of common assessments, such as code review, security testing of IoT devices, desktop apps, internal corporate networks, etc., are skills that can hardly be found in a crowdsourced pen testing platform, while they are obligatory for a traditional pen tester to possess.
When talking about more complex assessments, such as product security testing, SAP pentesting, Active Directory security testing, or security reviews of cloud environments, these are skills that one can rarely find, if at all, in such service providers. If the client insists on hiring crowdsourced pentesting vendors, it’s 99.9% guaranteed they will get a subpar test and report.
Lack of control of the pentesters profile
When working with a traditional pen testing vendor, clients have more control over the selection of cybersecurity professionals assigned to the service, ensuring that they work with qualified and experienced individuals that fit the profile they’re looking for and have the necessary certifications they expect. Frequently, customers ask for the profile of the pentesters that will work on the project, sometimes even interviewing these individuals before signing any agreements.
Excessive pentester rotation
The crowd is not fixed, with new testers being onboarded and offboarded regularly, and the fact that projects in PTaaS platforms are usually assigned on a mix of first come-first serve basis combined with limited or no due diligence on the skills means there is an excessive rotation of pentesters looking into your scopes.
While bringing a fresh pair of eyes to the game can be useful, it also has disadvantages that are often not talked about. We heard from an existing customer in the financial sector, who had previously hired two of the most popular PTaaS vendors on the market, that the excessive rotation was negative because every time a new pentester came in for an engagement, they had to spend most of the testing time ramping up knowledge about their environment, which was more complex than average, wasting precious time needed for the actual pentest.
What initially was all fun and games and excitement to leverage the power of the crowd ultimately meant subpar findings, hardly any discovery of critical vulnerabilities, and a lot of frustration for 12 months for the folks behind the organization’s application security program.
Our proposed “solution” to this problem was to assign a semi-dedicated pool of pentesters from our team (all of them full-time employees) to always perform pentests for this client, increasing actual testing time and improving the overall quality of the findings.
Data protection, GDPR issues and confidentiality problems
Not every organization is comfortable with the fact that their pentest data (that can include personal data and other sensitive information) is in the hands of a freelancer sitting in a country with lax data protection regulations.
Most crowdsourced penetration testing platforms heavily rely on talent located in developing countries, which are often jurisdictions that do not have strict data security regulations.
Additionally, Europe-based companies might not be OK with the possibility of data transfers from their EU-located cloud servers to a freelancer located outside the GDPR-protected areas inside the European Union.
Conscious companies understand the items above can jeopardize confidentiality agreements and put user data at risk, making them more prone to avoid crowdsourced pen testing providers.
Employment contract breaches and conflicts of interest
Recently, a well-known PTaaS vendor posted on social media that 70% of their crowdsourced pen testers have day jobs, many of them employed as penetration testers for other organizations – whether consulting companies offering pentest services or internal application security teams.
Valid questions arise here: how many of these crowdsourced pentesters are in breach of the non-compete clauses of their work contracts with their main employer? Or worse, how many of them might be employed in the internal security team of a competitor and will gain temporary authorization to hack into your network?
Many companies hiring crowdsourced pen test services don’t seem to care about the source of the pentesters assigned to their projects; however, organizations with stricter compliance requirements might want to avoid the risk of using unvetted personnel that can even be employed by the competition.
Questionable quality control
Oftentimes, crowdsourced penetration testing doesn’t offer the same level of consistency and standardization in the security testing process. The expertise, methodologies, and thoroughness can vary significantly between testers, leading to potential gaps in coverage and inconsistencies in the identification and reporting of vulnerabilities.
These factors make it challenging to ensure quality control in crowdsourced penetration testing compared to other methods.
Liability insurance and crowdsourced testing
Professional liability insurance plays a crucial role in protecting organizations against potential damages resulting from a cybersecurity incident. With crowdsourced penetration testing, obtaining adequate liability insurance can be challenging, as the pentesters are not employees of the organization. Traditional penetration testing, conducted by professional firms or in-house teams, generally provides more robust insurance options, offering organizations better protection.
Lead time to start a pentest
This is where crowdsourced penetration testing companies do very well. Any traditional pen testing vendor worth its salt will be fully booked for weeks, perhaps even months in advance, as there’s a limit to the number of projects it can execute at a time.
Crowdsourced security companies, on the other hand, have an elastic capacity using their pool of freelancers, meaning they can have a very fast turnaround time to start an engagement. Some even claim they can start a pentest in 24 hours, which is quite impressive.
It’s true that such vendors can start your pentest tomorrow, but it’s also true they’ll throw in anybody available for the job – regardless of whether the pentesting team members are qualified for the assessment, well-versed in the tech stack or took their time to understand your unique business logic.
Pentest delivery platform
Here’s another area where PTaaS and crowdsourced security vendors excel. Most traditional pen test firms are still making their staff write a report in Microsoft Word and deliver a PDF document to the customer. For many pentest providers, the delivery method did not change at all for the past 20 years.
Many organizations have no problem getting the pentest report in a PDF, subsequently stashing it on SharePoint, and waiting to pick it up again before an audit or when sending the issues for a developer to fix.
However, it is easy to see why this old-fashioned model has a lot of friction and is not compatible with the speed of DevOps, making it less suitable for tech companies and organizations that write a lot of code. They would certainly benefit from a more seamless approach integrated into their development process – something pentest-as-a-service platforms offering crowdsourced pen tests understood well and knew how to adapt to this need.
Looking for a pentest provider? Let us challenge your cyber defenses.
Talk to our experts for a custom quote
Advantages and disadvantages of crowdsourced penetration testing
Crowdsourced penetration testing vs. bug bounties
While both crowdsourced penetration testing and bug bounty programs leverage the collective expertise of the cybersecurity community, they serve different purposes and fit into different stages of a company’s security maturity model.
A bug bounty program is a more open-ended initiative where the organization invites security researchers to find and report vulnerabilities in exchange for rewards. The unstructured and always-on nature of bug bounty often results in the identification of more diverse and unforeseen vulnerabilities.
This does not mean that bug bounty can replace traditional pen tests. Rather, it is a complementary activity that can be highly beneficial in a mature security program. However, relying solely on bug bounties could result in a reactive, rather than proactive, security stance.
In short, a balanced and comprehensive cybersecurity strategy will ideally incorporate a blend of penetration testing methods and a bug bounty program, adjusting the mix according to the organization’s security maturity level and specific needs.
Conclusion
While crowdsourced penetration testing can be a cost-effective method that offers speed, it’s important to recognize the potential risks it carries. From quality control issues to increased exposure of sensitive data and potential regulatory complications, these challenges can offset the benefits of the crowdsourced method.
Traditional penetration testing, performed by a dedicated team of professionals, offers a more controlled, consistent, and comprehensive security assessment. It ensures that sensitive data remains protected, provides better compliance with legal and regulatory requirements, and comes with robust liability insurance.
Meanwhile, bug bounty can complement these strategies by adding an extra layer of security and uncovering a wide range of vulnerabilities in external-facing assets. However, these programs are not a replacement for systematic penetration testing but rather an additional tool in a mature security program.
