Low throughput with www/apache22 on FreeBSD 8.1

Few days ago I installed FreeBSD 8.1 and compiled Apache 2.2.15 from port www/apache22. I tried to transfer some files over the internet with rather good connection (100Mbps in the weakest point, but several hops resulting in RTT ~27ms). But I got only 10 – 15Mbps throughput from FreeBSD on hi-end HW. I also tried to install FTP server (ftp/proftpd) but I got the same result. It was not a network issue or a client problem, because I had full speed transfers from another machine in the same network. What the fuck?

Step one… Sysctl parameters. The most fucking evil parameter is net.inet.tcp.inflight.enable. Disable it! Actually I tuned more parameters relevant to TCP/IP stack and kernel internals:

/etc/sysctl.conf

kern.ipc.maxsockbuf=16777216
kern.ipc.nmbclusters=32768
kern.ipc.somaxconn=32768
kern.maxfiles=65536
kern.maxfilesperproc=32768
kern.maxvnodes=800000
net.inet.tcp.inflight.enable=0
net.inet.udp.maxdgram=57344
net.inet.udp.recvspace=65536
net.local.stream.recvspace=65536
net.inet.tcp.sendbuf_max=16777216

OK, transfers over FTP were then running at 100Mbps. Fine. But HTTP file transfers were still fucking slow. Locally it was two times slower than FTP and much more slower over the internet (HTTP: 4-10 Mbps vs. FTP: 100Mbps).

Step two... Turning off sendfile(2) in Apache HTTPd.

This is also incredibly fucking option and there is no documentation pointing at it... Even Google does not know what to do when the Apache fucks up on FreeBSD 8.1. Well, I added one single line to /usr/local/etc/apache22/httpd.conf:

EnableSendfile off

And that's it. Both FTP and HTTP are running at full speed.