So my site was hacked. How was your week?

Friday night is usually a time to relax and revel in the fact that I get two days off from work. I do work in IT so the idea of ever being completely off the clock is a complete fantasy of course, but it’s a nice fantasy.  Unfortunately last night was anything but relaxing. I’ve had a cold the last couple days, so I haven’t been feeling great, but last night I felt worse when I discovered that this website had been hacked.

Thursday morning I received a tweet from a friend pointing out an issue related to my daily TechEd Europe countdown tweet. Part of the tweet is an image that includes text saying how many days are left in various languages. Instead of showing the correct characters, the image was showing question marks. I had to get to work, so I did a quick check and also realized that the RSS feed for the site wasn’t working and hadn’t been for several days.  What happened the day it broke? A PHP upgrade by my web host. I had done some testing, but apparently this was missed. A quick edit to .htaccess to correct the AddHandler and AddType directives and it was fixed.

Since I had made the same changes for testing on this website as the TechEd Countdown site, I figured I should make that same edit to ,htaccess. So I did. As soon as I reloaded the page to verify everything was still working, however, it was obvious that something was wrong. None of the thumbnail images that appear for each post on the main index were showing up. Check the Media gallery in the WoprdPress dashboard. Broken images there too. I reverted the changes I made to .htaccess, but it had no effect. Something was wrong, but I was now running late to work so I chalked it up to the PHP change and said I’d look at it later. Mistake number 1.

That night I was feeling under the weather (remember that cold I mentioned?), so I went to bed early and said I’d look at the site on Friday. Mistake number two.

It’s now Friday, I’m about ready to leave work, and an email shows up in my personal mailbox from Google Webmaster Tools. I glance at it and here is what I see:

That can’t be good. I read the message, and it wasn’t. Google had caught one of my pages redirecting viewers to a malware site. As protection, my site was now one of those that clicking on it in a search result would warn you that the site isn’t safe. At this point, I realize my image issue probably had nothing to do with the PHP change, it was probably due to whatever had been done to my site.

As soon as I arrived at home, I head to my PC and start investigating. The first thing I looked for were changed files. A file scan for files modified since the image problem popped up showed only a couple files had been modified (none of which should have been changed). I opened the functions.php file for my theme and sure enough, there at the top, was a big chunk of obfuscated PHP.  Here was the proof the site had been compromised.

The next thing I checked was the list of users in WordPress. As I feared, there was a new administrator account that hadn’t been there before. This discovery set off a chain of password changes:

1. I deleted the malicious WordPress administrator account.
2. I changed my WordPress password.
3. I changed my webhost password.
4. I changed the password for the MySQL database user for the site.
5. Deciding the previous step wasn’t enough, I created a whole new database user and deleted the old one.

New database user means it is time ot edit wp-config.php with the new credentials. While I was in the file, I decided to generate new secret keys for WordPress which invalidates all cookies and forces anyone who is logged in to reauthenticate. If you ever need to do the same, WordPress provides a link to generate new secret keys.

Turning back to the changed file, I checked the other few files that had a recent modified date and found the same code. I had a hard time believing that only these few files had been changed, so I ran a scan of my entire webhost folder for a section of the code. After the scan completed, there was good news and bad news.

Good News: Only my main Ladewig.com site had been compromised. The other sites contained on this host were all clean. So it would seem they came into the site via the site itself and were more or less confined there.

Bad News: Every single PHP file on my site contained the malicious code.

But wait, none of the files all had modified dates that were what I would expect them to be. Looking at the files more closely showed that the modified dates looked normal. The file creation dates all indicated the files had been created recently. Thursday morning. At the exact time I was editing the .htaccess file. The images broke right then and there because they were hacking the site at that very moment, replacing my files. Editing one or two files is OK, but editing an entire WordPress installation plus themes and plugins? NOt doing that.

Thankfully I had just made a backup recently in anticipation of the PHP change my webhost was making.  That backup made it relatively simple to replace the compromised files with clean copies.  A scan after the fact showed that everything was back to what I’d expect. I even downloaded a fresh copy of WordPress and compared the two to verify that my files were clean. Afterwards I reviewed various hardening guides for WordPress such as the Hardening WordPress page at WordPress.org, to make sure my permissions were set properly, blank index.php files were in place to block directory listings, and added some other fixes like restricting executing PHP code from the uploads folder.

I deleted all of the themes that weren’t being used. There were still themes in there from my original WordPress setup and which aren’t updated anymore. It’s possible someone could hide something in there. You just don’t know. Better to get rid of it.  The same goes for plugins I’m not using. If they’ve been deactivated for months or even years, it’s unlikely I’m ever going to use them again. Toss them out and eliminate another means to attack the site.

I also added a plugin called WordPress File Monitor Plus to keep an eye on my WordPress installation and alert me if there any changes to the files. it’s an older plugin, but it works as advertised.  It only scans once an hour, but if I had known about the hack an hour after it happened instead of after 34 hours as was the case this time, I’d have been better off.

At this point, I still don’t know how they got in. In the process of cleaning up my site I did find two folders that weren’t created by me and that contained some additional obfuscated code. At least one predated this incident, so it’s possible they gained access via a problem in an older version of WordPress or some plugin. I removed those folders and I’m hopeful my site is secure again. Just to be safe though, I think I’m going to look again at rebuilding the site from the ground up with a clean WordPress install and bring my content over to make sure something isn’t hiding in some dusty corner.

Once the site was clean, I requested a site review in Google Webmaster Tools and now my site isn’t wearing the cone of shame anymore. I’ve never been a big fan of Google, but I do give them credit for alerting me to the issue on my site. FeedBlitz gave me a warning too about the issue, but only after it had already been fixed. Nice try guys.

Remember the broken picture issue? Turns out the attacker changed the default content folder to point to a new folder they created. As a result, the links that were being generated for the thumbnail images didn’t point to the correct place. Restoring my configuration fixed that.  Next time I’m not going to dismiss something like the broken images as quickly as I did and investigate a bit more quickly.

So that was my Friday. How was yours?