Linux Performance Benchmark - Apache/Nginx
Since I’ve come to no solid conclusion in my last benchmarks, I chose to make this one web focused which makes plenty of sense since it’s my main goal. I was going to test apache and nginx with 3 kinds of pages:
- A very light page - “Hello world!”
- A normal page, with some text and a few images
- A heavy page with a lot of content and heavy background processing
I didn’t even get to benchmark all of them since I was able to make a final judgement before that happened. Keep reading.
Benchmark
I used ab to perform the tests. There were 3 different configurations:
- 10000 requests, 1000 concurrent
- 100000 requests, 1000 concurrent
- 100000 requests, 10000 concurrent
My research is primarily focused on scaling Rails (and everything behind it) and that’s why I overkilled the webserver with something like 10k concurrent requests.
I also performed the requests locally since I wanted zero network overhead - just pure raw webserver performance.
On a side note, neither Ubuntu nor Debian accepted my changes to /etc/security/limits.conf in order to increase the maximum files opened by a process. The default is 1024 but since I wanted to benchmark more than 1024 concurrent requests, the webserver needed to be able to open more sockets - 10000. Either way, I overcame this problem using ulimit.
Edit:I made the mistake of not mentioning package versions. So, we have:
- Apache:
- Gentoo - 2.2.14
- Ubuntu - 2.2.12
- Debian - 2.2.9
- Nginx:
- Gentoo - 0.7.62
- Ubuntu - 0.7.62
- Debian - 0.6.32
In order to be really similar, I’ve tweaked apache’s and nginx’s configuration on all three platforms. No big deal, most of it was making sure the modules loaded were the same.
Apache Results
10000 requests, 1000 concurrent
Ubuntu lagged behind and Gentoo won this round.
A non-obvious detail that should be mentioned is the extremely low time response of Gentoo in the worst case: 46% of Debian’s time and only 7% of Ubuntu’s. Now that I mention it, I wasn’t expecting Ubuntu to have this poor performance.
100000 requests, 1000 concurrent
Gentoo kicked Debian’s ass and Ubuntu wasn’t even able to complete the test, starting to refuse connections around the 90000th request. I’ve tried to run the test 5 times and Ubuntu kept failing with the same error, despite the obvious configurations being correct - I checked limits, apache and the kernel.
Once again, Gentoo seems to be more stable than Debian since the standard deviation of the average time needed to complete a request is smaller.
100000 requests, 10000 concurrent
Now this is something I wasn’t expecting at all. I rechecked Debian’s configuration (as mentioned above) and everything seems quite fine. I’ve ran the benchmark 5 times and all of them failed around the 5000th request, which leads me into thinking that it wasn’t coping with concurrent requests very well.
On a side note, I tried to push Gentoo into it’s limits by making 500000 requests, 20000 concurrent (ab’s limit). Everything went well, error-free.
Once again, Gentoo wins this round.
Nginx Results
10000 requests, 1000 concurrent
Finally, some different results - Debian wins the round but Gentoo is still the one with the lower standard deviation from the average request time - a clear sign of stability.
100000 requests, 1000 concurrent
Ubuntu falls behind the other two distributions. Debian and Gentoo show good results and the winner is clearly Debian. Gentoo keeps showing interesting stability signs by providing the lower standard deviation from the average request time.
100000 requests, 10000 concurrent
I wasn’t expecting this either - while Ubuntu had shown obvious signs of not being able to cope with so much concurrent requests, Debian was coping up pretty well, even beating Gentoo by a slight margin. Unfortunately, both failed this time. I reviewed, once again, their configurations and repeated the tests 5 times. Debian started refusing connections around the 20000th request just like Ubuntu.
Conclusions
I’m forced to conclude that neither Debian nor Ubuntu scale very well when it comes to webservers. Both failed when a high number of concurrent requests was initiated and therefore are ruled out of my future benchmarks. I was going to use other 2 pages for benchmarking, as I mentioned before: a page with some content and a heavy page. I didn’t even get to that point since Ubuntu and Debian ruled themselves out with this simple “Hello world!” test page.
Gentoo clearly won the Apache benchmark and also won the Nginx one, mainly due to it’s stability - Debian was actually slightly ahead before starting to fail. The fact it was outperforming the other two distributions might be related to it’s outdated version of nginx.
Bottom line: CentOS was ruled out because of initial misconfigurations (shown in my previous benchmarks) and, finally, Ubuntu and Debian were both disqualified for not being able to cope with a high number of concurrent requests.
Gentoo is the winning Linux distribution!
What now?
I’m going to make a head-to-head between Gentoo and the most popular BSD distribution - FreeBSD. I’ll go directly to the Apache/Nginx benchmark and see if it can keep up with Gentoo’s results. If it can, I’ll use the heavier pages I mentioned before in order to draw some solid conclusions.
After all this is done and I’ve finally chosen the distribution that I’ll use in future tests/benchmarks/development, I’ll be benchmarking many Ruby webservers - Mongrel, Passenger (on Apache and Nginx), Thin and possibly a few others.
