Web Framework Benchmark Description

To compare the performance of Snap to that of other web servers/frameworks, we ran some simple benchmarks using httperf: pong and file. In the pong benchmark, servers sent a 4-byte response of “PONG”. The file benchmark sent a 50KB PNG image file as its response. Our results and nodes are described below.

Most of the benchmarks were run with the following httperf command line options:

httperf --hog --num-conns 1000 --num-calls 1000 --burst-length 20 --port 3000 --rate 1000 --uri=/pong

All benchmarks were run on a dual quad-core Xeon E5345 2.33 GHz workstation with 8 gigs of RAM. In all these tests, we compiled Snap with “-threaded” and ran it with “+RTS -A4M -N4 -qg0 -qb -g1”. Using more than four threads caused the performance to decrease. This is certainly a good area for future improvement in Snap.

Disclaimer: We are not experts at optimizing these web frameworks. It is almost certain that the results for other frameworks are unfairly low. If you’re interested in the details, the code we used is in a github repository. We’re always willing to update our benchmarks with optimizations that don’t take an inordinate amount of effort to set up.

Pong Benchmark

The pong benchmark is quite simple, but there was a large amount of variation in performance. Here is a chart of responses per second (higher numbers are better):

The Numbers

It should be noted that the Happstack benchmark is not multithreaded. We compiled Happstack with -threaded and when run with “+RTS -N4”, the test never finished. When we aborted httperf, it reported a rate of 12.6 requests/second. We also tried running it with with the same “+RTS -A4M -N4 -qg0 -qb -g1” runtime options that we used with Snap. This caused a stack overflow.

File Benchmark

The file benchmark tests the ability to serve static content from disk. The results here were a little closer together than the pong results, but we still see the same general pattern.

The Numbers

In this test, we had more problems because Happstack does not support pipelining. So we switched to the following httperf line for Happstack:

httperf --hog --num-conns 100000 --num-calls 1 --burst-length 1 --port 8080 --rate 10000 --uri=/FiringGeometry.png

This means that Happstack’s results in this test are not really an apples-to-apples comparison with the other frameworks, but they’re still somewhat meaningful since they reflect a deficiency in the server.

See the raw test output for more detailed httperf output.