Turning Penetration Tests Into Sales Artifacts
If you find value in this story, please consider making a donation to the [HS]2 program. It prepares a group of first-generation and/or low-income students of color to succeed in college by empowering them with STEM-based skills, a family of driven peers, and a space to see the light and power in their own voices. Even $1 helps by demonstrating broad support to larger institutions considering donations.
Although not mandatory, external penetration testing is a great practice to engage in. For those who aren’t familiar with the term, penetration testing describes handing a product over to a group of experts skilled at using it incorrectly. No one wants their search field used as a spoon holder.
While many companies have internal teams dedicated to this very thing, customers would rather see this done by an external firm. It’s the same reason they want external audits like SOC 2.
These can be a great experience for everyone involved. Customers can feel better about the product, product engineers will feel better about what they’ve shipped and sales will have a great document as a discussion point with customers.
What Should Get Tested?
Anything precious! It’s largely a sniff test and impossible to quantify. Some good questions about any new feature could be:
- If this is misconfigured or used incorrectly, could it turn into a company-ending event?
- Does this new feature interact with any Critical or High risk customer data categories?
- Will this new engineering work serve as the foundation for multiple new features or releases?
Answering yes to any of those is a strong indicator.
Early Product Engagement and Clear Deadlines
Security should work with the Product Manager to establish a clear deadline when the feature is supposed to be in the hands of Revenue. This is super important. In all likelihood, Customer Success Managers have already committed to it being in the hands of a large/important customer by a certain date. It’s also important that a three or four week delay might cause Revenue to miss a quarterly sales target.
With the go-to-market date firmly establish, work backwards and ask product engineers when they’ll have the MVP in a state ready for testing. There should be no material changes to the architecture or implementation at this point, otherwise the test won’t be accurate and provide the confidence its supposed to provide. There’s no point in spending tens of thousands of dollars testing something half baked.
Kick Off Call and Setting Expectations
Schedule a kickoff call with your penetration testing firm as soon as possible. Their calendar is often booked months in advance. You don’t want to scramble to find a firm last minute, work with a firm you don’t know well or need to pay extra for a rush job. Smooth and predictable is the key.
The kick off call with the penetration testing firm should include:
- The Engineering Manager overseeing the project.
- The Technical Lead and perhaps one more key product engineers, but no more to keep the team lean.
- The Product Manager
- A Security Engineer responsible for consulting through the project, from the scoping of the test through the testing period and all the way to the end when all findings are fixed to their satisfaction.
- A project manager of some kind, which could be a GRC Coordinator, Engineering Operations person, the Security Manager or the Security Engineer.
The kickoff takes half an hour and should end with the penetration testing firm know exactly what is in and out of scope for the engagement, what code bases they need to review, what they are and aren’t allowed to hammer, which product engineers they’ll be working with, when the project must start so the launch isn’t delayed, and whether this is a black box test (zero feedback or assistance) or a partnership engagement.
Socializing The Outcome
It’s critical that the product engineers and the product manager socialize the testing start date, how long it will take, the fact that there will be findings that must be fixed, that those fixes could take a week or two (possibly more), and the fixes must be fixed before the feature is ready for release. Socialization is critical because Marketing, Revenue and Product leadership will be eager to get this out the door. Their expectations must be tempered with the reality of the long tail that comes with penetration testing.
Leading up to the test, the project manager should keep the testing firm appraised of the MVP status. The firm has blocked out these dates on their calendar, this is money they are expecting and it will be harder to shuffle dates the closer it gets to the actual testing period. Delays must be socialized to them as soon as possible.
Testing 1, 2, 3
A week before the test starts, the project manager should do the following:
- Create a private Slack channel with the people listed above and the testers. A good naming convention is 202N-QN-feature-description, subbing in the correct year and quarter numbers. This will make it easy to remember and searchable later.
- Create a folder in Google Drive, Confluence or whatever your document storage medium is using the same naming convention. This is where the testing contract should reside and all results from the firm should live going forward.
- Make sure the product engineers have shared all the relevant code and technical investment briefs.
- Ask the testers if there is anything they’re lacking. The last thing you want is to be paying them to wait around while you get them access to something or scheduling calls to sort out details.
Once the project kicks off, the product engineers should consider this their primary responsibility. They’ll largely be able to go about their normal days; it’s more like being on call for the occasional question. The last thing you want is for the testers to pose a question in the Slack channel and seen them wait a day, half a day or even a couple hours for the answer.
The testers should ask as many questions and engage as much as they’d like with the product engineers in pursuit of quality. If they find something that looks particularly scary, they should alert everyone immediately and not wait until the end of the week or the engagement. Testers should also provide a quick summary at the end of each week in Slack. It shouldn’t be too formal, but product engineers and product need some certainty that the engagement is on the right track.
The testing firm will produce a detailed report at the end of testing. The engagement is not over though! The product engineers should schedule a call with their consulting security engineer to review results with all the context they have as employees. All Critical and High findings must be fixed, no questions asked. No customer wants a feature with known Critical or High security flaws in it. Fixing Mediums is at the discretion of the security engineer working in partnership with the product engineers. It might be perfectly fine to fix this in a few weeks. There will be pressure from Product, Marketing and Revenue to hurry (it’s their job), but remind them that part of their product is the final customer-facing report.
Wait, a what?
It’s never a good idea to share an unredacted penetration report because that is an actual roadmap on how to hack the company. Customers will ask for it, though, and a one or two page summary strikes a happy medium. It’ll be a summary of the engagement, the scope, and a sanitized list of all the findings. An excellent way to kill a sales deal is for that report to show there are open Critical, High or material Mediums issues that aren’t fixed. You shouldn’t want to explain that to a customer.
Fixes Are Good For Revenue
One the initial testing is complete, product engineers will want to make their fixes as soon as possible. This is good for product and good for the testers, who will be moving on to another engagement. They should coordinate their fixes in the Slack channel and get them retested as soon as possible. Once all the Criticals, Highs and material Mediums are done, the product engineers should add all the open findings to their backlog and the tester should write up the customer-facing summary.
Once the test is complete, make sure the product engineers have revoked the tester’s access to the code and any environments established for the test.
Product, Marketing and Revenue should be given a copy of the customer-facing summary to use going forward. It’ll be a great artifact for engaging with customers.
Summary
Done correctly, a penetration test will make customers feel good about the product they’re using, even if it’s a little later than initially promised. That builds great trust with customers and your Account Executives or Customer Success Managers. And for engineers, it’s a great educational opportunity to learn about security and reduces anxiety about whether their code is safe. There’s never a guarantee. But external testing can get you as close as possible.
