Safire Knowledge Base

Performance Characterization

Performance Characterization uses application mix traffic profiles together with malware to simulate real-world scenarios for firewall performance testing.

A traffic profile is described by the following parameters:

• Number of users (IP addresses)
• Number of applications
• Number of sessions
• Number of TLS sessions
• Bandwidth allocation, server-to-client direction
• Bandwidth allocation, client-to-server direction
• Users per application
• Sessions per application
• Bandwidth allocation per application, server-to-client direction
• Bandwidth allocation per application, client-to-server direction
• Average packet size per application

By default, the test uses 10 iterations with gradual increase in traffic load, which includes number of users, number of sessions, and bandwidth, from one iteration to another. During each test iteration, the test sends 10 unencrypted and 10  encrypted malware traffic from the servers to the clients to  exercise firewall’s anti-malware engine. Between iterations, the test clears the firewall’s session table if such a command is provided. If not, the test will wait for firewall’s aging time before continuing to the next iteration. During each iteration, the test measures the seven key performance metrics:

• Throughput and goodput
• Latency
• Concurrent sessions
• Packet loss rate
• Malware passed by firewall
• Firewall CPU usage
• Firewall memory usage

Using this method, the test measures key performance metrics as a function of traffic load, which provides a better performance characterization of the firewall under test if compared to as a function of time.

Performance Characterization is different from throughput testing method in that the former increases both the number of connections and traffic rate, while the latter only increases traffic rate. It is also different from benchmark testing method where binary search algorithm is usually employed to find the a particular maximum capability.