TL;DR - If you have a great team you can overcome any obstacle in your way.
I woke up this morning with my usual routine: shower, coffee with my "God time" (as I tell the girls), and empty the dishwasher that ran last night. The girls were sleeping in late and I had a little extra time, so I decided to get an early jump on Qakbot to see if I could find out anything to help at work.
I donned by Googlian Monk robes and there wasn't anything new about Qakbot. Interestingly enough the news article ran in the local paper about our delimma was number one in the rankings. Given the fact that we're a rather "small fish" in the big picture, this was kind of interesting. It also enforced the fact that we were dealing with some issues that were fresh on the market.
I went back to Twitter, and there was only one entry after my own tweet regarding the issue. It had some japanese characters in the tweet with a shortened link, and I was wary, given my paranoia at the moment about how easily and fast this virus can spread. However, I was on my Mac, which the virus wasn't targeting, so I figured I could be a little more adventurous and right never EVERY little nugget of information helped. I clicked the link and it went to the Symantec website in Japan. I could barely read any of the characters on it, but amongst the japanese characters were the words "ACL" and "fix". This is good news! One of the problems we face with Qakbot is the fact that it goes behind the scenes and eliminates Symantec's ability to do its job. If you try to mess with it too much, the Qakbot will change the ACL (access control list) to the app so you can't touch it whatsoever. If the proper virus definitions were in place, we could use the ACL tool to gain access, update the definitions, and potentially clean it. I tried using Google's page translation service, but it didn't help much. However, we have a coworker that is fluent in Japanese, so I saved the link for him to look at.
This was good, but I wasn't quite satisfied with just one solution. So I decided to keep digging. I went over to the SuperUser site (part of the StackExchange network) and decided to type in Qakbot on a whim. There was only one question that came back, and it was regarding general use of anti-virus software. However, one answer (not even the top one) indicated that Microsoft Security Essentials helped them and it was the only one that had stopped Qakbot. The answer was dated in 2010, so there was a good chance they were referring to the 2nd generation version of the virus. I've been using MSE myself since it went out to the public and have been very happy with it. It is a small install, does not bog down your computer, and has caught numerous things for us on occasion. I realized that if we went this route we'd be ditching our current anti-virus software program, but we were looking for anything right now, and we have a MS license agreement and this might be covered under it.
Armed with this information I drove to work. I didn't bother going to my cube, I went straight to the "war room" where the Ops and Networking offices are connected. The first question I asked was if our Networking guy that speaks Japanese was in yet. He wasn't, but should be soon. So I waited. In the meantime, I talked with one of our ops guys and he mentioned that Symantec had just released a removal tool. I said "Really?! Because I just saw an ACL fix tool, but it was on their Japanese site!" He showed me the site he found and it was the same one I had found, but in English. It had a timestamp only a few hours ago, so his reasoning was that the solution came up overnight and Japan's website was updated first. That made sense. But this was exciting, we had an option to pursue.
While we were all talking about updates from yesterday, one of our Networking guys had mentioned that he had been running his laptop on Microsoft Security Essentials for nearly a week now, and hadn't seen the the virus at all. I couldn't believe things were linking up like this! I told him how I had just found a slightly old answer post indicating that MSE worked for another person and specifically mentioned Qakbot. I forget if I went back to the Ops guy I was just talking to or if he overheard us, because he then mentions that Microsoft has an Enterprise type of MSE out called Forefront Endpoint Protection. What's more, it is covered for free under our enterprise license and they had started tossing around the idea a couple weeks ago of testing it out and maybe swapping over to it.
This was precisely the momentum we needed. It was about time for a group meeting again and we tossed the idea out on the table. A couple other folks seemed keen on the idea, and we even had a few infected machines, plus a private network we could work with. You could tell there was some light beginning to shine again. Another one of our Ops guys was smart enough to note that we should continue to pursue the Symantec option in case this doesn't work. I'm excited about what a great team we have. We get to work setting up the "mini lab" and our ops guy give us a CD with the installer plus the latest virus definition and proactive scanning updates. This is another good sign because most likely we're going to need to be able to patch these systems (provided the solution works) in an offline mode since the second the network goes live, Qakbot likes to try to spread.
We get an infected XP machine up, remove Symantec, and install Forefront Endpoint Protection (FEP). We set it to do a full scan and realize this is going to take a while. It's going to touch every file, and this particular machine has a fair amount of files and not the most recent processor on it. In the meantime, our Tech Support guy that has done all the initial digging on the virus comes in and says he has a tool that will detect and remove it too. It's a free tool called "HouseCall" by Trend Micro, and it seems to catch the virus in about 5 minutes, as opposed to our ongoing process. The only downside to HouseCall is that it requires the machine to get onto the Internet in order to get it's latest definitions to scan with. This is very promising too!!!! We have three solutions in place now that we can go with, and now we're in the agonizing "waiting period" to see if FEP works, if Symantec can get back to us, and how HouseCall might fit into things as well. Since we currently only have an XP machine on the FEP scan, I run over and grab my own PC, because I have a good hunch it is infected, that is Windows 7 based, and start up the process as well.
After a while the scan finishes on the XP machine. Sure enough, it FOUND and CLEANED Quakbot! You could sense relief pouring into the group. The next big step was testing it against future infection. One of our Networking guys hooks up an infected machine to the private network and turns it on. Nothing happens at first. Our Networking guy remembers that the infected machine is looking for some kind of mapped drive vector to get through, so they setup a couple of mapped drives from the clean machine to the dirty machine and then start pinging it. Not even a minute into this our clean machine pops up an alert box. It saw Qakbot coming into it and removed it on the spot! Success!!! We keep the ping going to see if it tries to infect a second time, but oddly nothing happens. We look back into the log and notice that not only had FEP stopped the incoming virus, but since it was located on a mapped share drive, it went out to the other machine and killed it there. Double success!!! We now know that FEP is really aggressive against it. We still need to verify this software will work on our servers as well. We pull a server that has already been shut down and not as critical and get the software setup on it. We have our large meeting with the administrator, so we get it setup and then go to the meeting.
This group meeting is a lot better than yesterday. The reports of FEP's initial success is encouraging to those that haven't been there, and any questions about the program are easily answered. We're starting to talk about how to fully deploy this all of our machines. We talk a bit about HouseCall as well, but at the moment, the success of FEP and the lack of required Internet connection trumps the speed of HouseCall. Now talks about how to map out deployment get heavier. Ultimately this is a discussion that needs to be handled by a couple of folks outside this meeting, and they are designated. Our Ops guy that is working with Symantec asks how much longer to continue that route. He had been getting good feedback from them, but the directions they had provided wouldn't work after the first couple of steps due to how bad Qakbot had mucked up the system. He was waiting for some feedback, but wouldn't wait too much longer given the path we were on.
I was tasked with training the fellow programmers on how to do a clean of Qakbot using FEP. This process soon became ingrained into my head:
- Disconnect the computer from the network.
- Disable System Restore (we don't want to rollback into an infected state).
- Uninstall Symantec (reboot if it doesn't force you too).
- Install FEP.
- Install the updated DAT files (scan can't actually run without them).
- Fire off a full scan.
We used the programmers existing machines and the amazing thing was that with the proactive scanner in place, the trojan version of Qakbot was often caught by the proactive scanner before the full scanner even had a chance to run! Very good news yet again.
Now it was time for war. It was close to the end of the day, but I wanted to see this in action. I ran over with a Networking buddy over to the HR office, since it was right next door to ours. They have about a dozen computers or so, so this initial battle would be a good test case. A couple of other folks went over to the main adminisrator's office. Armed with three CDs, we really start to knock out this process on a "pipeline" basis. You boot a computer and disable the restore. Since this can take a minute or two, you go over and start the same process on another computer or two. Then you can come back and start the next step on another computer. By the time we've started the scan on all the computers, the first ones are done with their scan. Sure enough, they come back infected. One particular computer had a good 5 infections on it:
- Qakbot.genB (Trojan)
- Qakbot.!genB (Backdoor)
- Qakbot.arc3 (Backdoor) - I suspect the third generation.
- Qakbot.genB (Trojan)
- Qakbot.!genB (Backdoor) - these last two were in a separate location
It just goes to show how nasty this thing is. After cleaning these machines out, we set them to run a second full scan. We aren't going to plug them back in until we get a clean bill of health.
I "pass the rock" to my fellow Networking warrior. The battle went in our favor and we heard a similar story over at the administrators building. We call it a day early, because we know that tomorrow will be a lot of work and we'll be hitting a lot of machines to clean things out. However, the battle went to an outstanding team today, and tomorrow will look even better.