Book chapter: Software test methods for the CISSP exam

Software test methods and testing levels for secure application development are covered in this excerpt from Elsevier’s CISSP Study Guide.

CISSP Study Guide software test methods

In CISSP Study Guide, authors Eric Conrad, Seth Misenar, and Joshua Feldman discuss application development security, including software test methods such as fuzzing and combinatorial software testing.

The following excerpt comes from Chapter 9: Domain 8: Application development security (pdf).

Software Testing Methods

There are a variety of software testing methods. In addition to testing the features and stability of the software, testing increasingly focuses on discovering specific programmer errors that could lead to vulnerabilities which risk system compromise, including a lack-of-bounds checking.

Static testing tests the code passively: the code is not running. This includes walkthroughs, syntax checking, and code reviews. Dynamic testing tests the code while executive it.

White box software testing gives the tester access to program source code, data structures, variables, etc. Black box testing gives the tester no internal details: the software is treated as a black box that receives inputs.

A Traceability Matrix (sometimes called a Requirements Traceability Matrix or RTM) can be used to map customers’ requirements to the software testing plan: it “traces” the “requirements”, and ensures that they are being sent.

Software Testing Levels
It is usually helpful to approach the challenge of testing software from multiple angles, addressing various testing levels, from low to high. The software testing levels of Unit Testing, Installation Testing, Integration Testing, Regressing Testing, and Acceptance Testing are designed to accomplish that goal:

  • Unit Testing: Low-level tests of software components, such as functions, procedures or objects
  • Installation Testing: Testing software as it is installed and first opened
  • Integration Testing: Testing multiple software components as they are combined into a working system. Subsets may be tested, or Big Bang integration testing tests all integrated software components
  • Regression Testing: Testing software after updates, modifications, or patches
  • Acceptance Testing: Testing to ensure the software meets the customers’ operational requirements. When this testing is done directly by the customer, it is called User Acceptance Testing.

Fuzzing (also called fuzz testing) is a type of black box testing that enters random, malformed data as inputs into software programs to determine if they will crash. A program that crashes when receiving malformed or unexpected input is likely to suffer from a boundary checking issue, and may be vulnerable to a buffer overflow attack.

Fuzzing is typically automated, repeatedly presenting random input strings as command line switches, environment variables, and program inputs. Any program that crashes or hangs has failed the fuzz test.

Combinatorial Software Testing
Combinatorial software testing is a black box method that seeks to identify and test all unique combinations of software inputs. An example of combinatorial software testing is pairwise testing (also called all pairs testing).

NIST gives the following example of pairwise testing. “Suppose we want to demonstrate that a new software application works correctly on PCs that use the Windows or Linux operating system, Intel or AMD processors, and the IPv4 or IPv6 protocols. This is a total of 2 x 2 x 2 = 8 possibilities, but as the table below shows, only four tests are required to test every component interacting with every other component at least once. In this most basic combinatorial method, known as pairwise testing, at least one of the four tests covers all possible pairs (t=2) of values among the three parameters.”

NIST Pairwise Testing Example

Test case




















Download the entire chapter (pdf).

Reprinted with permission from Elsevier Inc. Copyright 2011. "CISSP Study Guide" by E. Conrad, S. Misenar and J. Feldman. For more information about this title and similar books, please visit the book’s page on the publisher's web site.

Dig Deeper on Employee Training and Development for MSPs