One of my pet peeves is poorly written penetration testing reports. I procure a lot of pentests, audits and security assessments so I see a lot of these reports. I also conduct exhaustive internal testing for my organization so I have a lot of experience writing them as well. The one thing that continues to blow my mind is so many consultants completely miss the point of the assessment. I am hiring you to evaluate my systems and provide documentation to me that demonstrates how effective security controls are and to help guide me down the path to make things better. Part of this whole process involves executive support for security investments, including the report we just procured! So why is it that these documents are written for only a technical audience and are written so poorly I’d need to scoop out some gray matter to stupefy myself enough to comprehend the report?
I’m starting a series of blog posts to look at some sample penetration test reports I find around the web, starting with this particularly bad one at http://blog.hackaserver.com/howto-complete-a-penetration-test-report/ that I found linked from a forum post at http://ethicalhacker.net as an example of how to write a good report. NO NO NO NO! This is NOT a good report!
First off, a sample report without a sample company provides zero context and implies that you really don’t know what you are doing. Look, I understand a sample report needs to be sanitized to protect the customer but if you can’t demonstrate your ability to map technical risk to business risk then I’m not going to hire you. One critical component of the executive summary is creating that mapping for the CEO so he can understand in business language he understands what the high level issues are.
Secondly, in the executive summary there are technical terms and phrases like “The system also lacks strong authentication credentials which puts most of the running services at risk.” Is an average CEO realistically going to know what that even means? You don’t identify specific missing controls in an executive summary. This is where programmatic and process issues, high level meta-issues and the mapping to risk items the CEO cares about go. If the CEO needs a translator for page 1, the report is a complete fail.
This report was rife with grammatical errors, improper tense and other issues. Let’s look at just the first couple sections of the executive summary (by far the most important part!):
Sample Report
The Executive Summary
Background
The client tasked with performing an internal/external vulnerability assessment and penetration testing of the Metasploitable machine. This system have been identified as extreme risk of security controls being compromised. The purpose of this assessment was to verify the effectiveness of the security controls put in place by the client to secure business-critical information.
Overall Posture
The system lacks effective patch management of the vulnerable services. The system also lacks strong authentication credentials which puts most of the running services at risk.
The first sentence is an incomplete sentence. “The client tasked CONSULTING FIRM with performing…”
The second sentence uses the word “have” instead of “has” and is a very poorly worded sentence that should either be removed or clarified to explain the reasons for “extreme risk”. I would personally just remove it as it adds nothing to the report.
The usage of “was” vs “were” is pretty basic and this should be changed. I also tend to capitalize or italicize the names of tools like Nmap in a report so it stands out a bit.
The overall posture section begins with a present tense statement that deviates from the past tense used until this point. At several places throughout the document this tense changed back and forth.
Lastly, the sentence I alluded to earlier with my second point. Immediately after this section the author does make an attempt to create that business level mapping by mentioning “catastrophic financial losses” but appears to be an assertion that is not backed up with any mention of the function of the system or how it creates impact or even what “catastrophic” means. This is only page 1 of a 23 page report. The rest of it is no better.
Most of the rest of the document is simple tool output with a sentence or two describing the results in a seemingly random fashion. Tool output does not belong in reports folks. That’s material for an appendix. Sometimes a screenshot of a compromised system is all you need. Instead there are 19 pages of tool output when all that was needed was a single screenshot or possibly a Visio diagram and a paragraph or two explaining how it was done and how to resolve the issues.
It’s not helpful for the reader to have to wade through 23 pages of report for a document that could have been done in 5 pages or less. Get in, communicate results to the appropriate audience in a way that makes sense for them and move on. Short and sweet.
