A Tale of Stale Content

Engine room, by Maggie Stephens
“Engine room”, by Maggie Stephens (Pot Noodle) on Flickr. Licensed under CC-BY.

The low-attention-span “I-just-want-this-problem-fixed-now” version: if you’re having trouble with Nginx serving up stale static content, and you’re running it in a virtualised environment, try setting sendfile off; in the configuration file (either under that server, or set it globally under http).

Abstraction. It’s something we take completely for granted in the computer world. Even as I write this, there are countless processes going on in my computer, beneath the browser into which I type. I don’t, and shouldn’t have to, understand all of them to get this blog post written.

Sometimes a problem comes up that is just weird. It seems completely illogical. But these computery things are supposed to be nothing but logic, right?

When we eventually arrive at the solution, after many hours of hair loss and bad language, we are reminded of the sheer complexity of these systems. Our assumptions about how something at a higher level should behave are entirely dependent on the premise that the lower levels are all doing exactly as expected too.

It’s humbling, in a slightly odd technical sense. We all need to be humbled sometimes.

The Problem

We’re working on a shiny new workflow for development and deployment of the websites we build. This new way of working employs various tools including Git, Puppet, Capistrano (in its most spiffy multi-stage configuration), Vagrant (going hand-in-hand with VirtualBox), Nginx, PHP-FPM and Varnish.

While working inside a Vagrant development box, some changes were made to a static file — a CSS stylesheet, in this case. Nothing unusual about that.

On reloading the page in the browser, the file hadn’t changed. Caches were cleared, caches were even disabled, but Nginx still served us up the old file. Even more oddly, the ‘last modified’ timestamp would update to the last time we saved the file, but we’d still be served the stale content.

So, we moved away from Vagrant — and tried to reproduce the problem on our new Nginx + Varnish web server. The same result — change the file, but get served stale content, even when all the caches had been brought out of the equation.

The Solution

By this point, I had decided that it was likely Nginx that was the culprit. After all, we’d practically eliminated every other part of the system that could be the problem. Every great story has the perfect opportunity for a montage. Well, at this point, imagine me going through the Nginx config file, line by line, to an appropriate problem solving tune.

I stumble across:

sendfile on;

Hmmm — interesting. I wonder what it does?

It turns out that it makes Nginx invoke the sendfile() feature of the Linux kernel to get the file from the disk. It’s lower overhead than the alternative, which is for Nginx to ask the kernel to do a read() and then a write().

Except that, apparently, sendfile doesn’t work so well in virtualised environments like VirtualBox, where our development Vagrant box runs. It doesn’t seem to work properly under Xen either, the environment that our hosting provider Linode uses to give us a virtual private server.

I thought I had eliminated every other part of the system except Nginx. I hadn’t.

Every assumption I was making depended on the truth that everything underneath Nginx was behaving exactly as expected too. It wasn’t. This issue was caused by a peculiarity in the kernel of the virtual machines.

Switching that setting to sendfile off; instantly solved all of our woes. Maybe we lose the most minute modicum of performance — but that’s what all the other layers of caching are for!

From Frustration, Forward

Despite the frustration caused, I actually think this was a really fascinating problem. Again, it is humbling, in the sense that it reminds us at how many different levels we are standing on the shoulders of the tech that came before.

It underlines my belief that to be really good at IT problem solving, you need to have (at least a basic) end-to-end understanding of a system.

Computers aren’t magic. When they work, it might seem like it. But when they don’t, you get an opportunity to begin to break that illusion down, piece by piece. You’ll start immensely frustrated, but, each time you get to that resolution, you end up a little more enlightened.

7 Comments »

  1. Tzn wrote:

    Saying that sendfile doesn’t work well in virtualized environments is slightly misleading. By the sounds of if the problem is that shared folders isn’t working so well, because if you had the file in a guest only directory it would work right?

    Comment — October 27, 2012 @ 6:59 am
  2. Tzn wrote:

    Saying that sendfile doesn’t work well in virtualized environments is slightly misleading. By the sounds of if the problem is that shared folders isn’t working so well, because if you had the file in a guest only directory it would work right?

    Comment — October 27, 2012 @ 2:59 am
  3. Chris Van Patten wrote:

    That’d be safe to say except that we also encountered the same problem in our Virtual Private Server, which is a virtualised Xen instance. With that in mind I think it’s safe to say this is a problem that is more about virtualisation than about the shared folder setup (we don’t use shared folders on our VPS).

    Comment — November 29, 2012 @ 9:33 pm
  4. Chris Van Patten wrote:

    That’d be safe to say except that we also encountered the same problem in our Virtual Private Server, which is a virtualised Xen instance. With that in mind I think it’s safe to say this is a problem that is more about virtualisation than about the shared folder setup (we don’t use shared folders on our VPS).

    Comment — November 29, 2012 @ 4:33 pm
  5. Thanks for this, was having lots of issues in the Virtualbox VM I’m using locally and this this has solved the stale content issues.

    Comment — March 7, 2013 @ 7:57 pm
  6. Thanks for this, was having lots of issues in the Virtualbox VM I’m using locally and this this has solved the stale content issues.

    Comment — March 7, 2013 @ 2:57 pm
  7. Ben Di Maggio wrote:

    Thanks for this thoughtful post. It’s rare that a blog post both helps me solve a problem (stupid Apache on VirtualBox had the same problem; with your help, I realized that the fix was to set “EnableSendfile off” in httpd.conf) and provides useful food for thought. Humbling indeed!

    Comment — September 4, 2015 @ 1:03 pm

Leave a comment