Key findings and conclusions:

Download the PDF

• The NetDevices Services Gateway software converted a Sun Fire X2100 server into a multi-services gateway that outperformed the Cisco 3845 in throughput comparison testing
• In side-by-side testing with the Cisco 3845 ISR, the NetDevices Services Gateway software shows superior sustained network performance as multiple services are enabled
• Superior performance was demonstrated in packet forwarding, ACL processing, and NAT functions with higher throughput compared to the Cisco 3845
• NetDevices Services Gateway software demonstrated the ability to upgrade discrete software components without requiring a reboot of the device

NetDevices of Sunnyvale, CA engaged Miercom to independently verify the operation of the NetDevices Services Gateway (SG) software, and the Cisco 3845 ISR.

The Miercom performance testing found the NetDevices Services Gateway software distinguished itself in several aspects: In the key result showing better network throughput with successively increased overhead of active network services, and over a range of packet sizes.

The Services Gateway software, running on a Sun Fire X2100 server, showed sustained balancing of branch office-level traffic, achieving 97.5% forwarding and ACL processing of local Gigabit Ethernet LAN traffic while maintaining a fixed 45 Mbps VPN WAN link.

SG Software demonstrated a “hot-swappable�, no-reboot upgrade. In addition, network service modules can be upgraded independently, for increased control of upgrade scheduling.

Verified Performance - This chart shows the change in network line rate as a percent of full throughput (Gigabit Ethernet connection), in the 1500B UDP packet test at a rate of 164Kpps forwarding, as successive core services are turned on: 100 Access Control List (ACL), entries, 100 ACLs + NAT, and finally the addition of QoS services to the previous two. The Services Gateway software maintains throughput consistently in comparison.

Competitive Testing Note: The tests and test methodology that produced these results were proposed by, co-developed with and/or influenced by the vendor sponsoring the review. Miercom assured their fair and accurate application. Consider all the pertinent factors in your product selection.

Test-bed Setup
Services Gateway Software Performance. The thoughput testbed (Fig. 1) was designed to compare key performance aspects of the NetDevices Services Gateway software and the Cisco 3845. The SG Software ran on a Sun Fire X2100 server, with 512MB memory.

All connections were Ethernet Gigabit, with a copper wire connection out of the SUT to a media converter switch, continued on a fiber optic connection to the SmartBit 2000 unit.
This was is the only side-by-side comparison of the suite of tests performed.

Throughput traffic was generated by a Spirent SmartBits 2000 placed in a loop with the SUT via fiber optic GigE connections. The ingress and egress connections from the SUT originated on copper, through a media converter switch, to fiber optic GigE. SmartBits generates only UDP protocol at gigabit rates.

Test traffic was composed of bi-directional UDP traffic over statically configured ports, with successive packet sizes of 64B, 512B, and 1500B. Each packet size was tested with the successive addition of network services, including the application of ACLs from 100 to 5000 in count, NAT, and QoS. The ACLs were defined so as to enforce the application of every rule on inspected packets. Test measurements were compiled and reported on the Spirent SmartApplications v3.00 software. The VPN WAN test was with the SG Software only, connected to a comparable NetDevices unit across a WAN.

For the SG Software, the ACL rules were applied in each interface of the unit, but in only one direction to ensure one-time packet inspection. For the 3845 Cisco unit, filters were applied on only one interface, but in both directions, achieving the same one-time inspection result. The configuration files for each test were reviewed for the command settings, validation of the IP address and software version info for each unit on each SUT test. The configuration files were saved, and immediately reloaded. The test performed was the SmartBits Acceptable Loss Throughput Test Results for length of 30 seconds. The results were the resulting � Avg % passed� for each of the frame (packet) sizes.

LAN/WAN IPsec Performance. The second testbed, he LAN/WAN IPSec testbed (Fig. 2), was used in this test and applied only to the SG Software on the Sun Fire X2100. The packet forwarding rate of unencrypted LAN IP traffic was measured while maintaining a fixed VPN over T3 at 45 Mbps with AES encrypted traffic. The SmartBits unit sent 1 traffic flow per interface, using two interfaces. Four slots total were used, 2 for each bidirectional flow, also copper to fiber optic, with one flow for unencrypted 1500B “local� traffic and another for encrypted 1400B IPSec WAN traffic maintaining a fixed 45Mbps bandwidth. Results were captured with the SmartWindows feature of Spirent SmartApplication.

Upgrade test (not shown). Two computers were connected across the SG Software server while maintaining a continuous ping between the two machines. The QoS module on the SG Software server was changed to a different version, while maintaining the continuous ping between the two computers. For this test, the Cisco reboot time and the time for the interface to enable was measured.

ACL 5000. 5000 ACLs were defined on the SG Software, configured as above, to ensure all rules are executed before forwarding, and compare the SG Software’s extended performance against executing 100 ACLs.

Results
Performance Test. In the 64-byte benchmark tests, the SG Software outperformed the Cisco 3845 in raw packet forwarding (no services running) by 45%+, 1096Kpps to 600Kpps. As documented in “How We Tested�, the same services were turned subsequently on for each unit for the performance testing: a traffic load generated, the results measured and reported for the diverse range of packet sizes of 64B, 512B, and 1500B.
The favorable comparison of the SG Software continued with the remaining tests, with the SG Software showing comparable or better throughput under the same conditions. The SG Software was measured at 165% higher throughput than the Cisco 3845 in the 64B packet size test cycle with “100 ACLs� to be exercised (24% to 9%) with the testbed 1Gbps Ethernet load. The SG Software compared favorably with services added to the test, with 375% higher output than the Cisco 3845 while running 100 “ACLs + NAT� (19% to 4%), and a sizeable 466% higher comparative throughput while processing�100 ACLs + NAT + QoS� (17% to 3%).
In the 512B test cycle, the SG Software maintained throughput higher than the 3845 even as services were added. With “100 ACLs� the Service Gateway software showed 75% higher throughput (98% to 56%), increasing to 260% with “100 ACLs + NAT� (98% to 27%), up to 444% of the 3845 throughput with “100 ACLs + NAT + QoS� (98% to 18%).
The trend of superior throughput for the Services Gateway software continued into the 1500-byte test. The Services Gateway software pinned at 99.8% throughput over the three service enablements in the cycle. The comparative output of the Services Gateway software increased from relative parity with the Cisco 3845 with “100 ACLs�, to 30% higher throughput with “100 ACLs + NAT�, to almost 87% higher throughput with “100 ACLs + QoS� enabled. In a separate testing combination of multiple services of “1000 ACLs + NAT�, a significant processing load over the three increasing packet sizes, the Services Gateway software measured respectively 19%, 98%, and 99% of full throughput. The 3845 ranged, in the same packet size series, with 4%, 27%, to 77% respectively, of full throughput.

LAN/WAN IPSec. This standalone test of the SG Software showed significant favorable performance in the forwarding rate of the LAN traffic, 160Kpps (97.5% full throughput) while maintaining a 45Mbps VPN T3 WAN connection. The LAN carried unencrypted traffic in continuous flow mode, with a 1500 byte packet in IP format. The WAN flow carried AES encrypted traffic also in continuous flow mode, with 1400-byte IP packets.

Live Upgrade. A version change of the QoS module of the SG Software was accomplished while maintaining a ping session between two computers attached to the SG Software server. No outage was detected, and no further reboot was necessary for activation. In comparison the 3845 shows a time of over two minutes for a required upgrade reboot and accessibility to the device interface.

5000 ACLs. In a demonstration of the processing capability of the SG Software, the throughput percent performance was measured with 5000 active ACLs, set up and measured as in the Performance Section, with no additional services enabled. All of the ACLs were processed before forwarding the packets. Comparing the results to the “100 ACLs� test, there was less than 1% change in throughput between the results using the 64B packet (24% of line rate) and the 512B packet (98% of line rate). The effortless handling of increased ACL processing from 100 to 5000 with minimal degradation points to the Services Gateway software’s strength in scalability.

Branch Office Scalability. The performance of the Services Gateway software coupled with the cost of the server hardware tested above, make a compelling case for a branch office router platform. By leveraging off-theshelf hardware costing under $1,000, (Sun Fire X2100, price approximately $745 as configured) NetDevices demonstrated a persuasive ROI advantage as compared to Cisco’s proprietary platform costing over $15,000.

Note: All publicly available materials from the competitive vendor, along with the technical expertise and judgment of the testers, were applied to ensure these vendors’ units were appropriately and properly configured for each test scenario. Cisco was invited but declined requests to provide Miercom with technical support for this testing.

Miercom Verified Performance
Based on a workout of the NetDevices Services Gateway software and review of its configuration and operation as described herein – Miercom attests to these findings:

• The NetDevices Services Gateway software converted a Sun Fire X2100 server into a multi-services gateway that outperformed the Cisco 3845 in throughput comparison testing
• The Services Gateway software showed superior sustained performance as multiple services are enabled in side-by-side testing with the Cisco 3845 ISR
• Superior performance was demonstrated in packet forwarding, ACL processing, and NAT functions with higher throughput in comparison
• NetDevices Services Gateway software upgraded discrete software components without requiring a reboot of the device

About Miercom’s Product Testing Services…
With hundreds of its product-comparison analyses published over the years in such leading network trade periodicals as Business Communications Review and Network World, Miercom’s reputation as the leading, independent product test center is unquestioned. Founded in 1988, the company has pioneered the comparative assessment of networking hardware and software, having developed methodologies for testing products from SAN switches to VoIP gateways and IP PBX’s. Miercom’s private test services include competitive product analyses, as well as individual product evaluations. Products submitted for review are typically evaluated under the “NetWORKS As Advertised™� program, in which networkingrelated products must endure a comprehensive, independent assessment of the products’ usability and performance. Products that meet the appropriate criteria and performance levels receive the “NetWORKS As Advertised™� award and Miercom Labs’ testimonial endorsement.