TL;DR - There are some common patters to how Qakbot infects a machine. Once you understand them, you can quickly make work of them.
It is a Saturday morning, and I'm typically not going to the office, but today is different. With the Qakbot virus on the run, we've pulled another "all hands on deck" type day to really flush this thing out. We meet at 8 in a conference room and everybody is on time, ready to rumble. The director gives us the rundown on what has worked, what initial feedback has been from the various departments, and he starts to take a bit of feedback from various team members on what has been done and/or what needs to be done. Unfortunately this starts to drag out a bit longer than it should be. I feel a bit like a ninja now. I have my tools mastered. I can move quickly and efficiently from computer to computer, killing of Qakbot wherever I see it. I've gained a pretty in depth knowledge, so that I can typically surmise an infected machine vs. a clean machine within the first few moments of my encounter with it. I also know how to get behind it and eliminate it when it tries some of its tricks.
Eventually we get to the end of the meeting and all the Tech Support folks move to one side of the room. We are supposed to pair up with one of them and get to work. My "padawan" from yesterday (now a full fledged Jedi Master) and I immediately look at each other without a queue. We want to check up on our location from yesterday. We find the Tech Support guy that has that as his primary station, give him the rundown on our idea (since we also need to check across the street and want to do cleanup of our current building) and he buys in on the plan. This looks to be a good day, we can already tell. He's very much a "let's rock this joint" type thing as well, so the ninja clan of 3 set off to do our thing.
The first stop is across the street to followup on a Networking guy's scans from yesterday. Unfortunately the "master" key that the director has given us isn't all that masterful, and so we head out to our location from yesterday. We're excited to see that all of the machines, sans one, are clean, minus the two that had already started a second scan from yesterday. Ironically enough we were talking with the owner of the one infected machine and they were rather proud in the fact that they don't do a lot of extraneous surfing or anything on their work laptop, so they were pretty sure they didn't have it. That said, Qakbot likes to "surf" through your network shares and other internal connections, and given their higher position of authority within the department, they were more likely to be accessing files from the remote shares, and then to get it.
Using our ninja stealth to quickly clean up this location, we "call it in" to the "clean leaders" so they can come by and do a second approval. The policy has changed over night and now the "clean leads" will double check a location before calling the Networking guys to turn on a VLAN. I'm not sure why this change is in place, but it stands to help out the Networking folks from getting bombarded from multiple locations. I do feel bad for the clean leads though because it is two guys and they will be fielding A LOT of calls today. They also have most of the keys, so we have them meet us at the location we tried to get into before. They have a good set of keys and we make quick work of the location, noting only two machines that need a second scan run on them. I'm looking at my sheet and I'm finally crossing the line where my clean machines are outnumbering my dirty machines that I've had to work with. Just below the floor we're on is the "vault" and a room where old records are kept. We get down there and I find my first "odd" system. It's a rather large tower case hooked to a HUGE machine named "Taylor". I have no idea what the device does, but it looks important. And wow! The tower is running a dual Xeon processor setup. Pretty spiffy. However all three admin passwords don't work with it. We're at the point where we are to remove the "red tagged" systems so they can be worked on at the Tech Bench, but this one is unique. We put a big note on it and I figure I'll talk to the gal who is normally in charge with this area on how to proceed.
With two smaller buildings under our belt, we join the fight at our second largest location for our organization. There are three floors to it as well, but some of them are behind some really tight security, so we have to meet up with the main Tech Support person there to get our marching orders. This was also the point where you learn a valuable lesson about teamwork and how to evaluate what really matters in a situation. When a crisis occurs such as this, there are often a lot of "cooks in the pot" with the best method of how to do things. By now, me and my fellow ninjas have a great system in place on how to work things out. At this location, however, the Tech lead has a different set of rules they want to roll by. They want two full scans, even after the first one comes clean. There are devices to attach, but only after the first scan comes clean. A few other oddities like that. We receive the word and get to work, and don't bother to argue. What's the point in that? We need to slay the Qakbot and there's more than one way to skin a cat. I modify my methodology a little to accomodate for their wishes, and find a happy medium.
We quickly get to our ninja magic again and discover that the Networking lead is over here kicking the pants off of Qakbot too. I like this guy, we've done a lot of brainstorming together through this process, so it's like greeting a fellow warrior on the battlefield after a long hiatus. We swap some updates about where the networks are at while we go through the machines. In no time we've cleared the remainder of items for the room, and head over to the conference room on the first floor. We come in to a room full of laptops. We have some mobile folks in organization, and they require ruggedized laptops with no CD-ROM drives. I find our new programmer there, and give him some works of encouragement and appreciation for coming in on the weekend. Our department secretary is there, and she's starting to slay Qakbot like it's nothing. EVERYBODY is doing what they can. The new guy indicates things are a bit slow going since they only have one USB CD-ROM device to hook up to the laptops, and that's where the ninja magic occurs again. The three of us pull out a USB drive or two each (always come prepared) and we slice through laptops like it is nothing. This is where pipeline architecture REALLY starts to shine. We have a row of a dozen laptops that we literally just walk down a table, prepping each step. I think we have all of them starting to scan within 20 minutes.
My former Padawan and I go up to the second floor to help with some other scans in the heavily locked down areas. It's a little eerie, and quiet, since nobody is around. Nearly all of the machines have Qakbot, and the Tech lead for this area laughs and says most of the folks here have free time on their hands. We continue to finish off this area and a few others, and it appears that all of the floors have been scanned and we're waiting for the results. A lot of us are hungry, including myself, for which even the pizza I brought earlier didn't satisfy the hunger, so we take a break for about 30 minutes and get some grub. It's good to break, and eat with folks I normally don't see on a day to day basis. The comment arises about how well we've all worked as a team, crossing beyond our normal duties to make things happen, and how great it is. It is unfortunate that it takes something like this for it to happen. I make a mental note that if I ever do move up into a management type position, I need to figure out a way to make team bonding this effective without the stress of patching 1500 machines and 150 some odd servers over the span of 4 days.
As we finish lunch, my body starts to protest going back to work. This was expected, and I laugh to myself about it. I start to get into the groove again and all is well with the ninja arts. The team of 3 ninjas pick up a fourth (the Tech gal that normally is in charge of the large building we cleaned yesterday) and we go to the other main building of the other Tech lead. Wow, that sounded vague, but just go with it. This building hasn't been touched yet, and this is a perfect time to see how well the four of us can meld and get things done, since there are a large number of machines. I won't get into the nitty gritty, but we work amazingly together and get the whole building up on its first and second scans by the time 7:00 PM rolls around. I think it took the four of us about 3 hours to do it, and I could see it taking a team of 6 longer than us. There was one area we didn't have keys to access, but that would be tackled tomorrow. Oddly enough I found one machine that Symantec had caught one variant of Qakbot on with the initial boot up, but I still had to flush the system out. It must have gotten some updated definitions before the infection.
The four ninjas head back to the main building exhausted after a very productive day. Tomorrow is Easter, and I've already talked to the director about taking tomorrow off since we have a lot of family plans. He tells me not to worry about it, and I really appreciate his understanding in this matter. Some folks in his shoes could have been really stubborn about it. It'll be a good break and on Monday I think we'll be able to finish things off and start to look at what "normal" work looks like again.