Stacking the Odds in Your Favor: How to Choose the Best Web Application Penetration Test Partner
So, you are in the market for a Web Application pen test. Or is that a security assessment? Maybe it’s a vulnerability assessment or a compliance audit? The terminology is dizzying. Adding to the mess is that each of these things may (and often do) mean different things to clients and vendors.
One vendor’s definition of a security assessment may match another’s pen test, which may not fit the client’s definition. So how do you know that you’re getting what you need, and can you tell if the vendor is giving you high-quality work? While there is not one single thing that can provide that insight, there are a lot of clues that you can get from your potential vendors to give yourself some sense of assurance that they are the right vendor for your needs.
This article aims to arm you with some questions to ask potential vendors. In addition, it guides you in identifying a quality vendor based on the questions they ask you so that you can find the right vendor offering a suitable assessment for your current needs.
Questions to ask yourself before searching for a vendor
Knowing why you want or need a pen test is perhaps the most crucial piece of information you can gather to identify the right vendor and ensure that the work aligns with your goals. Most of the time, I find that vendors fall into one of three main drivers.
The first driver is a new product or major release. Perhaps the client wants (or needs) to be proactive about identifying risk, or this release is in response to a prior breach or incident. Interestingly, many clients don’t have recent (or sometimes ANY) security objectives. While a deeper discussion of defining good security objectives is beyond the scope of this article, knowing what your application and your users should and should not be able to do will help the vendor test with those objectives in mind. Note that this type of testing strongly benefits vendors with deep expertise. These are usually the smaller, boutique firms that employ a small number of deeply technical experts who generally focus on this type of work.
The second driver is a compliance need. All too often (but thankfully not always), the client is more interested in satisfying a checklist of standard requirements than identifying unique risks and advanced attack techniques. If you are looking to satisfy a minimum base set of requirements, your need for deep technical expertise in a vendor is generally a secondary consideration. This is generally a sweet spot for larger consulting firms with automated tools, high volumes of audit and compliance consulting, and a deep bench of junior technical staff who often do the bulk of the work.
The final driver we experience is the “my boss/board is demanding we have a pen test.” As a vendor, this is often the most challenging client because they usually are fairly early in the security maturation process, are reluctant to welcome the process, and often have not considered any actual budget to address the issue. I suspect that if you’ve read this far already, you aren’t one of these clients. From a vendor perspective, the first thing we try to determine is “why do they want the test?” Often, with a bit of digging, what these clients need is less of a detailed technical test and more of a high-level risk assessment involving the scope. We also try to address the question, “What value will they be able to retrieve from the test” to ensure that we are delivering the right service with results that they will be able to digest and put into action.
So how do you determine that you've found the right partners? Of course, past experience is always a good indicator of future performance, but what if you haven't worked with them before? Recommendations and referrals from others can be helpful, but it doesn't consider the test type and the recommender's environment or needs.
We have found that, in general, if you ask the right questions and if the vendor asks you the right questions, the right partner for your current needs will become apparent. So what questions do you ask? That's a good first question.
Questions you should ask the vendor (and why)
- How do you scope this work?
Everyone has their own magic formula for determining the level of effort on a security assessment. Most vendors will likely not want to share details for obvious reasons, but you are looking to ensure that they have a clear and logical process.
If the vendor is unwilling to discuss how they scope a project, that points to a cookie-cutter approach or (even worse) a lack of knowledge.
- What are the skills and experience of the team performing the test?
Knowing who will be working on the assessment is a very telling factor. In general, larger firms will include a single senior resource and a large number of junior resources. It creates an opportunity to scale across projects while maintaining profitability, but usually at the cost of test quality. In addition, such teams rely heavily on automation and linear testing “runbooks .” Both of these items ensure consistency but often lead the testers to overlook the obscure and technical vulnerabilities that sophisticated attackers find and exploit.
A strong partner will have more experienced and senior resources focused exclusively on your project for the duration.
- How much of the testing will use automated tools vs. manual tradecraft?
This is often a tricky question. Automation has a valuable place within the lifecycle of a penetration test. Still, most of the time and effort for a good test will inevitably be spent manually interacting with the application and the APIs. Be concerned if the vendor’s explanation of their testing methodology does not emphasize manual interaction with the application.
- What can I expect as a deliverable?
Ultimately, the deliverable is the critical piece of a test. It’s the “proof” of the work. It helps you build your remediation planning and is often a requirement for verifying to your partners and clients that you take security seriously. It should contain results that are easy to digest for a non-technical audience and details not only on each finding but on suggested remediations. Finally, it should be in a format that is easy to index and ingest into your tracking systems.
A good security partner will be sensitive in the messaging and in selecting the most critical items to balance them with the items you are likely to be able to digest and remediate in a reasonable amount of time.
Questions the vendor should ask you (and why they ask)
- How often has this been tested in the past, and by whom?
I always ask this question for two main reasons. First, the answer allows me to understand if we are likely to find low-hanging fruit or if we need to plan on more advanced attack vectors. In addition, if the answer is “yes,” I will request access to the prior report(s) to ensure we retest the remediation of the findings.
Also, the frequency, number of previous tests, and number of findings all allow me to infer things about the client’s security maturity, which in turn leads me to ask a more advanced set of questions.
- What are your most significant security concerns?
This question carries a lot of data that we use to assess the client. The direct value is confirmation that the client knows what they should worry about, implying some level of overall security.
Second, it provides valuable context for our testing. Armed with this information, our engineers will ensure that we test for those concerns in addition to the tests we include based on industry, size, and known actors.
Finally, the detail, speed, and contents of the answer speak to the internal maturity of their threat model and risk assessment processes. This is valuable because it's a strong indicator of the sophistication of recommendations the client is likely able to use.
- Can we have (limited) access to the source code?
Since GRIMM engineers all have a development background, access to source code is a vital component of our testing. Top-of-the-line vendors will not only rely on dynamic testing; they will be able to review suspect source code, identify the specific code that enables the vulnerability, and provide detailed suggestions on remediation. This capability leads to more efficient and more effective testing and speeds up your remediation efforts.
- How do they rate vulnerabilities that they find?
Understanding the methodology and reasoning behind how a finding is evaluated for risk allows you to ensure how the vendor aligns risk with your organizational approach.
In general, we see vendors who rely on common frameworks such as CVSS, but other methodologies, such as EPSS and VPR, are also in general use. The primary consideration is that the tool allows your vendor to include and focus on the factors that are more relevant to your security goals and individual needs.
While hardly an exhaustive list, the considerations above should help you determine if the vendor you are speaking with has the right skills, approach, and operational perspective to provide the testing your application needs.
When it comes to this type of expert work, the adage, "you get what you pay for" most certainly holds true. Knowing your security objectives and how to evaluate your potential vendors' technical expertise and testing methodology will help you make the right decision for your current needs. It also provides some context in a range of prices and timeframes you will likely receive.
GRIMM experts can help you create a custom penetration testing package to suit your needs. To schedule a consultation or to learn more, email: info@grimm-co.com.
