Yet another lighttpd vs apache?

With my business partners I’ve been working on couple of web projects. One of those is starting to grow and I’ve decided to do a bit of testing. I’ve heard that people talk about lighttpd being quite faster than apache. Well, each single test I’ve seen was false. That’s why I decided to do my own testing. First part of the tests are static web sites and how usable are high traffic web sites in virtualized environments.

First mistake people do is testing ‘which app will do X requests first’ (you know those tests ‘ab -n X -c Y’?). Well, let me tell you right away – in those tests Apache will most probably be the slowest thing on the planet (particularly if you are using prefork MPM). Cause of its design, Apache, in a way, speeds up during time (it starts child servers or threads, depending on MPM). Lighttpd will win Apache any time in those tests. Then, people often do benchmarks on localhost – come on, what kind of a test is that?! Oh, right, you can achive 70MB/s throughput on localhost interface. Is that something you can get in the production? Those situations are possible only in local networks, but then again, 70MB/s throughput will require a gigabit network and hell of an load :). Then we have tests where people benchmark one site. So, when you put that together, you get ‘ab -n X -c Y http://localhost/some_static.html’. Right… Your visitors will visit only that page and then leave your web. Doh… If you are creating server and services strategy based on these numbers, you are better of with magic 8-ball.

So, let’s take a look at these graphs:

apache_vs_lighttpdapache_vs_lighttpd2

I’ve took the wrong approach of benchmarking lighttpd, apache prefork and apache worker against localhost. The numbers are fantastic, particularly for Apache – you see how easy is to crush lighttpd myth? Let’s take a look at siege output and some interesting numbers:

Apache Prefork MPM:

Transactions:              188781 hits
Availability:              100.00 %
Elapsed time:               60.53 secs
Data transferred:         5207.92 MB
Response time:                0.04 secs
Transaction rate:         3118.80 trans/sec
Throughput:               86.04 MB/sec
Concurrency:              138.70
Successful transactions:      188782
Failed transactions:               0
Longest transaction:            9.19
Shortest transaction:            0.00

Lighttpd:

Transactions:               74353 hits
Availability:               98.57 %
Elapsed time:               47.49 secs
Data transferred:         2020.14 MB
Response time:                0.12 secs
Transaction rate:         1565.66 trans/sec
Throughput:               42.54 MB/sec
Concurrency:              184.27
Successful transactions:       73860
Failed transactions:            1077
Longest transaction:            9.15
Shortest transaction:            0.01

Now, most of the people will look at Transaction rate and make a decision based on those numbers. But, one should realy look at Throughput; which is insane. That troughput isn’t possible in common production environments. Yes, there are situations where you can get more, but that’s <1% of (if at all) available web sites. With that throughput, transferred data is also big. 5GB in one minute just doesn’t sound reasonable. But the most important thing in this test (and it happened every time I tested) is that lighttpd doesn’t have 100% availability. And the response time of lighttpd is also disappointing. But, if you really want to do the test, you should, based on throughput and transferred data, figure out that this test is – wrong.

Tests should always be done from another computer which will be in the same network from which your clients will access your server. Localhost tests are OK only for web developers – they can see how much their PHP/Python/Perl application is slower than static HTML (they will look at average response time). If you are a sysadmin, that just doesn’t provide enough information to you.

In these tests, I compared Lighttpd, Apache Prefork and Apache Worker in two different situations. In one, web servers (including website) were hosted on Ubuntu Intrepid 8.10 (Dell PowerEdge T300/6GB RAM/Intel X3323), and in the other situation, web servers were hosted inside KVM on that same server, with all four cores. On real hardware, all servers provided ~400 requests per second (remember the 3000/1500 req/s from the localhost – you see how that test is broken?). But, inside KVM, results were worse – ~145 requests per second.

hardware_vs_kvmIt’s not a network stack in KVM, since it’s easy to achive >10MB/s (100mbit network) when downloading single large file. But, with lots of little static files, all three give 5-6MB/s. For those same files 11MB/s is standard on real hardware. Until I do more tests, I can conclude that virtualized web servers are OK for low traffic web sites (5-6MB/s is ~45mbit/s link), but if you are going to have high traffic site, you should really put it on a dedicated servers. I’m eagar to test dynamic content in virtualized environment.

Tests were done with default installs of lighttpd, apache2-mpm-worker and apache2-mpm-prefork packages on Ubuntu 8.10. Software used for testing was siege, while file list_of_urls was list of ~150 urls (gif, pdf, html, doc, jpeg…); basically copy of www.amzh.hr.

9 Responses to “Yet another lighttpd vs apache?”

  1. ntx says:

    Well, tho other benchmarks are often flawed they atleast include:

    - Hardware used
    - OS
    - config files of each app
    - size of testset/filesize or even the files for download
    - more than one stresstest app, ab/siege/htperf
    - keepalive on/off

    Thus this benchmark is as useless as most others out there, i´ve no way to reproduce it, so its just blabla…not even version numbers are provided.

    just my 2ct,
    ntx

  2. Maybe you didn’t read whole post? I’ve said which hardware is used, which OS is used, what kind of configuration and I even said what was tested website.

    Anyway, don’t get me wrong. This isn’t about lighttpd vs apache. It’s about KVM vs. hardware setups. First part is only about wrong benchmarks people often do – testing a single site against localhost. I wouldn’t mind having 1500 req/s – that’s great (that’s 5,4 milion req per hour) for a single server.

  3. leonel says:

    Can you benchmark the same with http://www.cherokee-project.org ??

    Thank you

  4. I could, but what’s the point? On real hardware, throughput is limited by network, and on KVM by something else. That’s the topic of the post – why is KVM more than 2x slower? I was expecting it being ~90% of hardware’s results.

    And those benchmarks on cherokee website are also flawed – they also do benchmarks with a single file. That’s just wrong. That’s pissing contest, without a real value. Test against random files (hundreds of them) and test on multicore systems. Those are real values, with a real hardware used in production – and I don’t favor any of web servers, I’m just tired of false benchmarks people are doing.

    Everybody should test their own website and their own network setup, with their own hardware and then decide which is better for their use. There’s no single winner in this one.

  5. Yann says:

    Hello Ante, what network NIC are you using for your KVM guest? How many CPUs has the guest? What version of KVM are you running?

  6. Hi Yann

    I tried with default one (realtek) and with e1000. Results were identical. KVM host was from Ubuntu Intrepid, version 72, iirc.

  7. Yann says:

    Hello Ante, could you try with virtio? The performances I get are a lot better with it.

  8. Besides what Yann said (use virtio), I was wondering what you’re using on the other end of the virtual nic. A bridged setup with tun networking? VLANs? routed configuration? I guess that could make some difference too… I’ve only used bridged tun devices so far and local performance seems to be excellent…

  9. I tried virtio, but I get same results. It should be better, hm, hm… On the other end is bridge with host’s Intel e1000 NIC.