TL;DR - Microsoft Forefront Endpoint Protection has proven it's worth as a successful Qakbot removal and protection tool. Time to give the virus it's due!
I get into work a couple of minutes early, ready for the task ahead. It will be simple, but tiresome. I'll be hitting as many machines as I can to get Symantec removed, MS FEP installed, and get the first scan kicked off. Along the way I'll probably be answering questions from some of the employees we encounter. As the digital warrior, I'm armed with three CDs with the software, sticky notes, two pens, a log sheet, directions (already memorized), and a set of stickers to mark machines that have finished the process.
Green means clean. Red means dead. Yellow is mellow, so we try not to use those...
I find the Networking guy that I hit the HR office with last night and we go back to survey our battlefield. Most of the machines need a second run, and there's one machine that the CDROM isn't working on, so we flag that for somebody else to peek at. After the HR run, we find that 95% of the machines can go through the process we've outlined, so those that can't we simply leave alone or mark with a note, we'll come back to those as time allows. The building we are in probably has the largest concentration of computers across three floors, and if we can get this entire building "touched" today we will have made excellent progress. We tag a couple of the computers that are clean and start up scans on the rest of them. It's 7 in the morning, and a few HR folks are already trickling into the office and we give them the rundown on things.
With the server folks working like mad on the servers, and the networking guys smart enough to have laid out our infrastructure with a nice VLAN structure, the goal is to get all the machines in a given subnet disconnected from the machine and then plug them back in on when they are green. Once we have a few machines clean and online, the networking guys can bring up that particular VLAN (which is still isolated from everybody else and the Internet) and then can start keeping an eye out on it to see if any odd traffic starts to spike, which indicates the virus may still be alive. This also gives us the added benefit of a larger test bed before we spread out to the world at large. Our organization spans a huge geographical area, with some offices a good 45 minute drive away. We have folks stationed out there permanently, but they are going to need extra troops and we don't want to send a group of people out there if the solution proves faulty.
Sadly I've already found a machine with 5 copies of the virus on it. This thing is wicked ugly, and it just reinforces it. We finish the HR building and the Networking guy needs to tend to networking duties now. I "pass him the rock" tell him to keep fighting the good fight, and now it is a little after 8. My fellow programmers are starting to trickle in, so I grab our new guy (he's only been here a week or two, what a way to start things out!) and we head down to the second floor, to a department with a lot of computers. I've worked with some of them during some special events, so sometimes it is nice to have a friendly face when you're bombarded with tech people fixing your computers. I notice that two programmers have already started down here, so the four of us look to have the perfect number for the amount of computers we have down here (I'm guessing about 25).
This is where my operating systems class in college really helps prove a point about the effectiveness of pipeline architecture. Instead of waiting for one machine to finish, I can boot up three machines, walk back to the first and log in and disable restore (which takes a minute), then walk down to the next machine while that's processing. In the grand scheme of things it takes less time, and as long as you don't put too many machines, or have to walk too far between them, you don't have much complexity. I start hammering away at this process and time starts to blur. I talk to a few of the employees a little bit while doing this process and all of them are grateful for the effort we're putting into this. They understand we're a little understaffed and have LOTS to support, and having all hands on deck is a testament to how committed we are to them.
Somewhere along the way scanning machines becomes second nature. I almost don't need my sticky notes to track the process for each machine, but it is still helpful for those cases where I get interrupted or a colleague comes by to help with the process (we have each others back). Soon the programmer from my office area that was in training come down, armed with stickies, CDs and, stickers. Her training is done and she's come to join the fight. I take her to be my "Jedi Padawan" and show her the ropes of scanning a system. We come across a couple of machines where the CDROM doesn't work and it's about time to find a solution. We start thinking about using a USB key, but are a little wary since we know Qakbot likes to jump to it. However, since the machines are off the network, FEP does it's job swimmingly, and we'll make sure to scan the drive before we pull it, this seems like a viable solution afterall. We run upstairs and notice that the "armory" of stickers, pens, and such also has some USB keys. We each grab one and I grab a couple older ones from my office and we quickly transfer the software over to them, and go back to the fight.
Eventually we get the area done and it's time for a late lunch. We do a sweep starting at the bottom floor to see if any of our other warriors need help and find a few miscellaneous machines that need tagging or a fresh scan. We can do this process so fast now that it doesn't seem like much to get one started. Doing all of these small tasks eventually leads up to a half hour or so and my stomach is starting to yell at me, so I force myself to go up and get some food.
There is some pizza left over from yesterday (somebody was awesome and bought pizza for us) so we run up to get some, and find out that the entire building is "touched" and a fair amount of them are clean. We've also discovered that the virus has really gone to town on others, having reset the administrator password or is way too stubborn to have Symantec uninstalled. Reports trickle in that disabling the service can help with the second issue, and I make note of it. In talking with my fellow warriors we discover that one of the smaller buildings that isn't too far away, hasn't been touched at all. My "padawan" knows some folks there, and we think there's enough time in the day for the two of us to go hit that building. We slam down our pizza and head over there.
Another blur in the space/time continuum arrives again as we get back to "fighting" mode, we each have a couple discs and these machines are a little slower, so we don't quite make as fast of progress as we probably thought. However, we've only worked about an hour past normal closing hours and we have all the machines fired off for a scan. I discover during this process that some of the Windows 7 machines like to hibernate themselves after 20 minutes or so. It baffles me a bit that an active virus scanning process is ignored as activity for the timer that shuts things down, but there's no time to investigate. I simply disable the power settings, leave the window for that up to remind me to swap it back, and let the system go about it's business.
We make it back to "headquarters" and find that the battle has gone really well today. The server guys have taken a huge bite out of the servers that need scanning and a handful of them are back up. The networking guys have been able to bring a VLAN or two back online and the results are looking good for that. The invasion has played out in our favor and we're confident that our solution is going to stick and we can apply it to the rest of the machines out there. We already know we're going to be working the weekend and there's a meeting at 8 in the morning, so I send a word of thanks and encouragement to everybody I can find and head home for a little bit of family time and rest. Tomorrow is another day and there's more qakbot to eliminate.