Belgian Testing Days 2013 - Tool selection: a successful approach by Bernd Beersma

by admin Email

Introduction

This is an article in two parts; first I will do a recap of Bernd Beersma?s talk about tool selection during the Belgian Testing Days. As an add on I will show how this process can be adapted to select open source tools as often used in security testing. I went to the talk just to see how other companies handle this process but I learned a lot of things and got a heap of good tips. So next time a company asks me to help them choose a tool I will certainly propose them this process ?

The talk

The slide deck can be found here.

Bernd showed quite convincingly that there is a process that can be followed that will deliver a high chance of success when selecting a new tool, the process is divided in several phases (thus far business as usual) and after each phase there is a ?go/no go? decision. This is the big difference between this process and the ones I saw being used in most companies. If after a phase there is not enough confidence that the tool(s) selected for the next round will be able to deliver sufficient quality then the process either goes back to the previous phase or stops all together (it would then start all over again with a clean sheet).


A poor selection of a tool can be caused by OR influenced by:

Poor business case

Lack of commitment

Poor automation

Cost


The 4 phases are a long list, a short list, a proof of concept (POC) and a pilot. It is important to note that Bernd tried to introduce some efficiency in the last two phases, the POC and pilot should be done on an actual environment so the time spend here is not lost if the tool is selected, in fact this will guarantee a ?running start? for the tool selected.

Next are some bullets of actions needed in each specific phase


The long list:

Identify stakeholders

Define goals

Create business case

Setup project team

Define general requirements (e.g. mind map)

Gather general information

Create long list (approximately 10 tools, possibly less)


The short list:

Define detailed requirements

Define priorities & weight for each requirement

Request for information (RFI)

Determine score (e.g. use a spreadsheet with the priorities and weights and score each line for each tool)

Create short list (e.g. top 3 tools)


The proof of concept

Draft POC requirements (i.e. an objective measurement of the POC)

Invite vendors for POC (shared involvement of vendor)

Execute & evaluate

Invite for a request for proposal (RFP)

Select the best suited tool for our purpose


The pilot

Basically same as POC but with just one tool/vendor

Select project

Define requirements

Invite vendor

Execute & evaluate project


The adaptation

When choosing tools for penetration testing we are often challenged by a myriad of tools, many of them small and single purpose. Often there is more than one small tool doing the same thing and choosing between them is difficult. Applying the procedure as above for all the tools needed for a penetration test would take too long. Also open source tools do not have vendors as such and RFI and RFP would not be possible. RFI will be replaced by searching the internet and trying to get an informed view and trying to replace a vendor, the RFP could be some kind of business case that estimates things like learning curve and cost, installation cost, maintenance cost etc. A great deal of objectivity is needed when estimating these costs since they can greatly influence the process. In most cases for a small tool the RFP can be dropped completely.

Most penetration testers use their own personalized system, many using open source tools or relative cheap tools (e.g. BurpSuite Pro), this means the stakeholders, project group and vendor related items can be dropped.

This leads to a slimmed down list of phases. I even put some estimates next to each phase.


The long list (1 day):

Define goals (since most of these tools are single purpose the general requirements cover this aspect also)

Create business case Estimate learning curve/effort

Define general requirements (e.g. mind map)

Gather general information

Create long list (approximately 5 tools, possibly less)


The short list (1 day):

Define detailed requirements

Define priorities & weight for each requirement

Determine score (e.g. use a spreadsheet with the priorities and weights and score each line for each tool)

Create short list (e.g. top 3 tools)


The proof of concept, pilot and implementation. These can be merged since most penetration test engagements do not last more than a couple of weeks.

Execute & evaluate in the next penetration test

Select the best suited tool for our purpose, possibly replacing another tool previously selected


Using this shortened method the total time spend on the tool selection should be significantly shorter while still keeping enough of the process to show other people how you came to that conclusion.

Trackback address for this post

Trackback URL (right click and copy shortcut/link location)

Feedback awaiting moderation

This post has 337 feedbacks awaiting moderation...