Beyond Security Blog | Which Is The Smarter Fuzzer? A Review of Beyond Security beSTORM® and Codenomicon Defensics®
Beyond Security - August 1, 2018

Which Is The Smarter Fuzzer? A Review of Beyond Security beSTORM® and Codenomicon Defensics®

Dynamic testing tools – feedback from a beSTORM® buyer

Software applications are by nature prone to vulnerabilities. Of course, software developers identify and fix functional or logical errors during the development phase itself. But as so many Zero Day exploits continue to remind us, many unknown software vulnerabilities do still manage to slip past the testing phase without being found or fixed, sometimes with disastrous results.

Conducting manual tests on any software with large quantities of programming codes is impractical. Tool-based, automated techniques are needed to manage this effort proactively. One of the most effective ways to identify software vulnerabilities by automated testing is the use of Fuzzing.

Fuzzing is a tool-based technique used to identify software bugs during the verification phase; this can contribute to identifying undisclosed security relevant bugs. To this end, the input interfaces of the target software to be tested are identified, to which targeted data are directed in an automated fashion while the software is being monitored for potential bugs. This makes it possible to prevent third parties from identifying vulnerabilities and thus from developing zero-day-exploits.

But there is no single Fuzzing method. Efficiency of each method depends on its creator. While one Fuzzing method works without having any knowledge about the data it is mutating while another method seeks to understand the application better so as to mutate its individual elements better. Select of fuzzers is, therefore, always based on needs of each application or operational environment.

For our heart rate monitoring device software, we wanted to test, analyze and locate vulnerabilities in its software. The idea was to fuzz our system by systematically sending invalid or unexpected inputs and reporting consequences of such actions, exposing software defects and vulnerabilities. Further, as our device connects with other systems on the network over Bluetooth, Ethernet or Wi-Fi we wanted a multi-protocol, environment-variable ‘Smart fuzzer’ that could understand overall application better and test individual elements dynamically.

Our fuzzer selection requirements were:

  • Exhaustive testing beginning with likely weaknesses and then expanding to every possible input variation dynamically
  • Extended Test range needed to include every field in entire protocols with every possible input variation
  • Certified ISA Secure with EDSA 1.0 & EDSA 2.0 qualifications
  • Re-create the attack with proof of concept code exported during test phase as PERL script
  • Existing software testing staff should be able to run tests on all protocols
  • Easy to understand – UI for the Fuzzer should be easy to understand by our test team

Although, there are more than 250 Fuzzers in the market, most are used to test web applications (25%), network protocols (45%), file formats (15%), Web browsers (10%) and APIs (7%). Only two multi-protocol, environment variable fuzzers are available in the market today; Codenomicon Defensics® and Beyond Security beSTORM®.

We set out to see for ourselves how Beyond Security beSTORM® compares to Codenomicon Defensics®. And this is what we found:

  beSTORM®Defensics®
Fuzzing MethodSmart Fuzzing
Model based fuzzing
Model Based fuzzing
Test TargetServer, Client, Applications, API, DLLServer, Client, Applications
Proprietary Protocol SupportSupported via customer or vendor development
Auto Learn capabilities
supported via vendor development only
Protocols Support250 (extendable)220 protocols
Test CasesThousands to billions per protocol
Users can also add test cases and Attack Vector type
Attacks are pre-defined
Recreating the VulnerabilityPossible to recreate using Perl Script without beSTORM® (Proof-of-concept code export)Possible to recreate using only Defensics®
Parallel TestSupportedSupported
Load TestSupported (maximum: 50 parallel threads and 250,000 attacks per second)not supported
Monitoringnetwork monitering, process moniter and an API for custom moniter supportnetwork monitering only
CertificationCertified ISA Secure (Embedded device and Systems Software qualification) EDSA 1.0 & EDSA 2.0Certified ISA Secure (Embedded device and Systems Software qualification) EDSA 1.0
Testing designExhaustive testing that begins with likely weaknesses but then expands out to every possible input variationSelected, highrisk, known protocol weaknesses
Test rangeEvery field of entire protocol is tested with every possible input variationSelected fields are tested for selected, known vulnerabilities
Test numbersMillions or tens of millions of teststhousands or tens of thousands
Monitoring capabilityProcess monitering, application ping, ICMP / ARP and custom monitoringProcess monitor
Testing of proprietary protocolsYes. ‘Learn’ function converts the BNF description used in RFC documents into attack languageNo
Supports native DLL callsYesNo
Full source code providedYesNo
Complete reporting on attacksExports proof of concept code in Perl script that re-creates the attack for easily repeating the failureNo
User interfaceModerate complexity, with typical training done in 1-2 daysHigh complexity, requiring extensive training
Cost of ownershipExisting software testing staff can run tests on all protocols of all productsHighly trained, expensive staff required for operation
Cost of purchaseMedian pricedHighest of all commercial, multi-protocol fuzzers
Custom protocolsNo additional cost optionAdded cost option
ScalableUse multiple processors or machines to reduce test timeNo

During our selection trials, Beyond Security beSTORM® showed higher standards of user-friendly handling and operation, fuzzing techniques supported and analytics and proved to be a superior Codenomicon Defensics® competitor. While both Beyond Security beSTORM® and Codenomicon Defensics® enabled us to reset the target application after system failure and implement the reproduction of bugs identified, only beSTORM® was able to generate a PERL script that recreates the vulnerability. Codenomicon Defensics® was definitely costlier. Although it supported a larger number of fuzzing techniques, feature utilization was difficult due its complex UI. We finally chose Beyond Security beSTORM® for our requirement as it scored higher on the following parameters:

  • Supported Fuzzing techniques and protocols
  • Analytical abilities
  • Software ergonomics
  • Extendibility
  • Documentation
  • Costs and license

Overall we chose Beyond Security beSTORM® as it was found to be the ideal Codenomicon Defensics® alternative for customers like us who are keen on a more flexible, cost effective, scalable and easy to use ‘smart’ Fuzzer.

Written by Beyond Security

We had an impossible mission: transform the hackers brain into a machine. Mission accomplished. Using automated software, Beyond Security is dedicated to finding common vulnerabilities and zero-day exploits at a fraction of the cost of human-based penetration testing. Businesses around the world have been relying on Beyond Security's vulnerability and compliance solutions since 1999. Whether you need to accurately assess and manage security weaknesses in your networks, applications, industrial systems or networked software, we're here for you - one step ahead of the hackers.