Init Crew

In my last blog post I wrote about problems with common web server stress tests. Even though I wanted to talk about KVM vs. hardware setups, it seams that most of that post is related to comparing apache and lighttpd. Since I didn’t do proper testing of lighttpd and apache, I’ve decided to do that now. I’ve tested performance of these two when delivering static content (html, jpegs, gifs, docs, etc…).

For start, let me introduce you to a hardware and software used. It’s a Dell PowerEdge T300 with Intel Xeon Quad Core X3323 processor and 6GB of RAM. Disks are in software mirror, but that’s not relevant for this test. Operating system used is Ubuntu 9.04 (still unreleased), 64bit version. Version of lighttpd in Ubuntu 9.04 is 1.4.19, while apache is 2.2.11. Tools used for testing were ab and siege. All software was acquired from Ubuntu repositories for 9.04, with no configuration changes (other than those mentioned in text). I did 6 tests with siege for every setup, removed the worst and the best score and calculated average on remaining four. With ab I did just one test, since those tests are, IMO, flawed anyway (ab by it self isn’t flawed, it’s just misued by lots of people).

AB

For my first test, I disabled three cores, so that I could simulate tests which most people do. In my last blog I explained why I consider them flawed. Seriously flawed. So, for first test I started ‘ab -n 10000 -c 25 http://localhost/index.htm’. index.htm is very simple file, size of 4KB. I used plain default configuration for lighttpd and apache2-mpm-prefork. Results are favoring lighttpd:

While these tests, IMO, don’t mean a thing, I went with a flow and did the same test with all four cores. For that test, I had to tweak lighttpd’s configuration. As mentioned at lighttpd’s wiki, lighttpd doesn’t like SMP very much. Among problems mentioned on this site, idea of corrupting access logs was horrible. Since this was testing anyway, I didn’t care about that, and access logs weren’t corrupted in the end. I’ve also did more tweaking, according to performance docs. So, finally, I added:

server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.event-handler = “linux-sysepoll”
server.network-backend = “linux-sendfile”
server.max-fds = 2048
server.stat-cache-engine = “simple”
server.max-worker = 8

to lighttpd’s configuration. Next, for Apache, I’ve enabled disk_cache module, just to level it with lighttpd, which does caching by default. Since Apache scales on SMP by default, no other changes were made to its configuration. I’ve also tested Apache Worker MPM. The test again showed that lighttpd is faster in serving 10000 copies of the same file:

SIEGE

When you are noticing problems with your current web server setup, it’s wise to have some statistical data of most visited sites. With that data you can create a list of URLs which you’ll use for testing your new setup. So, if index.htm is 20% of requests on your site, you’ll create a file that will have index.htm as 20% of its content. It’s also good to have some tool for testing which will randomize requests - I use siege for that. URLs in these file, in my test, were from couple of bytes to 0,25MB). In the absence of the gigabit switch, I used localhost for testing (most people do tests that way, even though that’s wrong, but let’s go with a flow). Again I removed three cores from my system and removed SMP tweaking in lighttpd. Results are:

As you can see, Apache worker with cache in memory is slower then lighttpd and Apache worker with cache on disk. I must admit, I wasn’t expecting that. Even Apache worker (without cache at all) is faster than Apache worker with cache in memory. But, let’s leave worker/mem in the dust, where it belongs. Even the fact that Apache delivered more requests per second than lighttpd isn’t quite important here. What’s really important isn’t shown in this graph, but in a ODS with raw data. Lighttpd didn’t finish single test. After ~40 seconds it just gave up serving content. I still haven’t figured out why this happens. But, anyway, I used the data collected while it was serving content.

Now, the same test with all four cores. I’ve again added support for SMP to lighttpd and again haven’t changed a thing in Apache’s configuration (except enabling cache). Results are similar to those of one core:

It’s noticeable that Apache worker with cache in memory scaled much better than lighttpd and even Apache worker with cache on disk. That was expected, since I even raised a bar of having 100 concurrent connections, instead of just 25 on single core. One could probably squize more req/s from Apache with tweaking of ThreadsPerChild and MaxClients.

In no way this blog was written to claim that one is better than another. It’s just that I’ll be needing static web server on new hardware and I wanted to have a solid data. At the moment, in my case, Apache seams better tool for the job. I might test cherokee and ngix too, just to be sure I’ve chosen the right tool.

Right, and still haven’t figured out why my virtualized provide so much worse results than hardware servers. :)

Raw data: raw-data.ods

With my business partners I’ve been working on couple of web projects. One of those is starting to grow and I’ve decided to do a bit of testing. I’ve heard that people talk about lighttpd being quite faster than apache. Well, each single test I’ve seen was false. That’s why I decided to do my own testing. First part of the tests are static web sites and how usable are high traffic web sites in virtualized environments.

First mistake people do is testing ‘which app will do X requests first’ (you know those tests ‘ab -n X -c Y’?). Well, let me tell you right away - in those tests Apache will most probably be the slowest thing on the planet (particularly if you are using prefork MPM). Cause of its design, Apache, in a way, speeds up during time (it starts child servers or threads, depending on MPM). Lighttpd will win Apache any time in those tests. Then, people often do benchmarks on localhost - come on, what kind of a test is that?! Oh, right, you can achive 70MB/s throughput on localhost interface. Is that something you can get in the production? Those situations are possible only in local networks, but then again, 70MB/s throughput will require a gigabit network and hell of an load :). Then we have tests where people benchmark one site. So, when you put that together, you get ‘ab -n X -c Y http://localhost/some_static.html’. Right… Your visitors will visit only that page and then leave your web. Doh… If you are creating server and services strategy based on these numbers, you are better of with magic 8-ball.

So, let’s take a look at these graphs:

I’ve took the wrong approach of benchmarking lighttpd, apache prefork and apache worker against localhost. The numbers are fantastic, particularly for Apache - you see how easy is to crush lighttpd myth? Let’s take a look at siege output and some interesting numbers:

Apache Prefork MPM:

Transactions:              188781 hits
Availability:              100.00 %
Elapsed time:               60.53 secs
Data transferred:         5207.92 MB
Response time:                0.04 secs
Transaction rate:         3118.80 trans/sec
Throughput:               86.04 MB/sec
Concurrency:              138.70
Successful transactions:      188782
Failed transactions:               0
Longest transaction:            9.19
Shortest transaction:            0.00

Lighttpd:

Transactions:               74353 hits
Availability:               98.57 %
Elapsed time:               47.49 secs
Data transferred:         2020.14 MB
Response time:                0.12 secs
Transaction rate:         1565.66 trans/sec
Throughput:               42.54 MB/sec
Concurrency:              184.27
Successful transactions:       73860
Failed transactions:            1077
Longest transaction:            9.15
Shortest transaction:            0.01

Now, most of the people will look at Transaction rate and make a decision based on those numbers. But, one should realy look at Throughput; which is insane. That troughput isn’t possible in common production environments. Yes, there are situations where you can get more, but that’s <1% of (if at all) available web sites. With that throughput, transferred data is also big. 5GB in one minute just doesn’t sound reasonable. But the most important thing in this test (and it happened every time I tested) is that lighttpd doesn’t have 100% availability. And the response time of lighttpd is also disappointing. But, if you really want to do the test, you should, based on throughput and transferred data, figure out that this test is - wrong.

Tests should always be done from another computer which will be in the same network from which your clients will access your server. Localhost tests are OK only for web developers - they can see how much their PHP/Python/Perl application is slower than static HTML (they will look at average response time). If you are a sysadmin, that just doesn’t provide enough information to you.

In these tests, I compared Lighttpd, Apache Prefork and Apache Worker in two different situations. In one, web servers (including website) were hosted on Ubuntu Intrepid 8.10 (Dell PowerEdge T300/6GB RAM/Intel X3323), and in the other situation, web servers were hosted inside KVM on that same server, with all four cores. On real hardware, all servers provided ~400 requests per second (remember the 3000/1500 req/s from the localhost - you see how that test is broken?). But, inside KVM, results were worse - ~145 requests per second.

It’s not a network stack in KVM, since it’s easy to achive >10MB/s (100mbit network) when downloading single large file. But, with lots of little static files, all three give 5-6MB/s. For those same files 11MB/s is standard on real hardware. Until I do more tests, I can conclude that virtualized web servers are OK for low traffic web sites (5-6MB/s is ~45mbit/s link), but if you are going to have high traffic site, you should really put it on a dedicated servers. I’m eagar to test dynamic content in virtualized environment.

Tests were done with default installs of lighttpd, apache2-mpm-worker and apache2-mpm-prefork packages on Ubuntu 8.10. Software used for testing was siege, while file list_of_urls was list of ~150 urls (gif, pdf, html, doc, jpeg…); basically copy of www.amzh.hr.