TL;DR - We've reached cleanup mode for Qakbot, and are going to analyze what happened, the good, the bad, and the ugly to better handle this situation next time.
Note: There is no day 6 since I had taken the day off for the holidays.
It's been a long weekend, and I'm feeling tired, but I know my teammates are feel far more tired than I am, since most of them came in yesterday to work on the infection. I walk in to the war room to get some updates and see if I can spread some light hearted encouragement to anybody that's there. It's going to be another busy day, hopefully not as intense as Saturday, but when you're body is tired, even small things can become big. I hear that we're probably up to 80% or 90% capacity with things, which is excellent news. I make a side comment about how everybody has been upgraded to slaying the virus, not merely killing it, and then say that maybe I should just call myself "Buffy the Qakbot Slayer" for the day as we finish things up. This draws some laughs from the folks there, and it inspires me enough to go make a quick name tag. The way I see it, if you can't keep things somewhat light amidst a struggle like this, you're not living life quite right. I make it, and immediately get a couple of chuckles and a couple of perplexed looks by some more teammates as they walk into the war room. This will be fun. I can get work done AND provide an indirect test to see those who are Buffy fans or not.
There's talk about having all of the programmers go over to the tech support office this morning to help man the phones while the tech support crew is out doing follow up. Networking turned on the internal network on Sunday so that folks can get back to their normal things. However, with some of the role changes that were made, and a little bit of confusion that follows bringing a major system like this online, there will be plenty of calls for help coming in. I'll be honest here, I'll take my marching orders as they come down to me, but I'd rather be in the middle of the action still. While talk of this and other things starts to bubble up with the incoming teammates, one of the Tech Support ninjas I was working with yesterday is heading down to the first floor of our building to follow up with one of the larger departments that was hit hard. There are a few red tagged machines that need to be sent over to the tech support office for formatting or password cracking, and their remote workers are bringing in their laptops to be scanned. I ask if she could use an extra set of hands, to which she replies "Sure." I grab my "slayer tools" and head downstairs.
We get downstairs and are greeted by a couple of members from this department. They are extremely grateful for what we've done over the weekend, and had a couple of general questions. One person looks at my name tag and laughs, the other looks at me a little funny, then at the co-worker who is laughing at me. As the ninja is answering questions and giving the rundown on what we're doing here, one person comes by and mentions one of the remote worker laptops are already in, sitting in an empty cube. I tell the ninja I can handle that while she does her rounds. I get the laptop fired up and scanning within a few minutes. The process is probably ingrained like muscle memory into my system now, and another two laptops come in towards the end of the process, which I make quick work of.
Once this is finished, I overhear a person asking for help about one of their applications, and walk over to take a peek. I know there are going to be some networking related issues since the public Internet is still cut off, and there are still exception cases to be made. With the ninja standing there as well, we watch the person fire up the app, and see the error message. It's rather cryptic, but we continue. The app loads and there's an area in the app with the typical 404 style error you find in Internet Explorer. The employee fires up a second application, more of a green screen terminal type program, and notes they can't get anywhere with it. Both of these look like network related issues, so I call up to the "war room hotline" that networking has setup and immediately get dropped into a conference call type setup. After checking with the guys up there, we quickly diagnose the issue as related to having external resources still shut off. They know the app I'm talking about and get that particular site opened up to traffic. In a couple of minutes they are looking good.
In the mean time, one of the programming managers comes down with a list in hand. He's working on getting all the red tagged machines put into a spot where it will be easy to move them out to the tech support building. With everything else in place now, I help him grab a few machines. We're putting them in a conference room just across the hall in one of the other department's areas. When I walk in the door with a machine, I'm greeted with thanks and told a couple of machines still aren't quite working. I put the machine in the collection spot and take a peek. One person just seems to need to wait a smidge longer for the network connections to re-establish. By the time I had returned their issue was resolved. The other needed to have their remote worker laptop scanned, so I make quick work to get that up and running. My programming manager comes in with a couple more machines, and he comments on how there were a few more over here, and one or two upstairs. It looks like we have our duty for the day: go around to the various departments, picking up red tagged machines and providing secondary assistance. That way the tech support folks can help their departments without having to drive around. I like this idea a lot, and the manager already has a couple of spots to hit.
I go and grab one of the official vehicles to load things into and we load up the car. We hit the tech support are and it's already loaded with programmers. It looks busy, but not extremely busy, so it was probably a good thing that I found other work to do, since I don't know how much I would have done. We get our marching orders from the tech support manager. The large building we hit on Saturday needs a few machines picked up, and then we were to go out to our third largest building (a good 20 minute drive away) and pick up/help the guy out there. There were still quite a few machines he couldn't touch due to locked doors over the weekend, and it is just him out there. We make quick work of building number two and head out to building number 3.
We spend the rest of the morning helping scan some machines out there (slaying never gets old) and working on a handful of laptops they take out into the field. Oddly enough about four of them present a weird message about a filter pack being needed in order to continue. Fortunately I remember hearing one of the tech bench guys talking to one of our Ops guys over the phone about that precise issue. I call up the Ops line, but the guy is busy. So I call over to the tech bench, and the guy there tells me he's looking through the directions to make sure the solution works, and then he'll call me. In the meantime, we go to check up on a few machines we have scanned. One laptop in particular had no CD-ROM drive, so I used my USB key and I could watch the virus jump right onto it the second I put the USB key into the machine and it popped up the explorer to view files. Two or three days ago this would have freaked me out to no end. Now I simply chuckle, knowing that the scan will flush it out safely.
It's getting to early afternoon, and we're starting to get hungry. We bring the tech support guy over to give him the rundown on everything, including the 4 laptops that remain with this filter issue. Like magic, I get a call from tech support. The solution works, but it's a file to download from Microsoft. Since part of this network isn't fully turned on yet, I don't have access to e-mail, and I don't have a smart phone. After pondering a moment, I tell him to e-mail me the file, and I'll make it work. The three of us head out to Jack in the Box for tacos (yes, they are awesome, trust me) and some more team bonding. When we finish, the manager and I decide to go back to tech support, drop off the machines we collected, and I can get the patch file needed for the laptops, then we'd head back out and patch them. Sure it seems like a lot of driving, but we're running the support line now, and that's what you do.
When we get back to to tech support. We drop off the machines, get the patch downloaded onto our USB keys, and hear some very encouraging news. By their estimates, we have ~95% of the machines in our fleet (1500) touched and cleaned. All of our servers (~150) are up and running, having been fully cleaned or finishing up their final scan. We've effectively turned out a solution over the course of 3 to 4 days (not counting the time to find one) and we're up to 80% capacity running. We're still sorting out some networking and permissions issues, but nothing that is disrupting the work of any department. This is very encouraging. All of the team deserved to be dubbed slayers, Jedi, ninjas, what-have-you. It took all of us to make it happen.
There's one final remote location in the next town with about 3 computers that need to be scanned. We figure there should be enough time to swing back by building #3, fix those laptops and start the scan, then head out to the remote location. We can start those scans off, since they take about 45 minutes at the minimum, and one of the tech support workers that lives out there can hit up the machines on their way home or to work. Ironically, or not so ironically enough, the last system we scan out there immediately shows signs of an infection, before I even start the full scan. I let the scan start, lock the machine, thank the employees there for letting us invade their space, and let them know somebody will be back tomorrow to scan things. We hop in the car to head back to the office and I figure I should have my manager take a picture of the name tag for posterity's sake. I've already started blogging about this "adventure" and there needs to be at least one picture in the series, right?
I get back to my cube with about 30 minutes or so remaining in the day. I lay down my weapons, having finally retired from my duties as a Qakbot slayer, but know I may get called into a couple small skirmishes over the next week (a warrior's job is never fully done). I'm now back into "normal programmer" mode again and take a peek at a couple e-mails. Sadly things have caught up to me by now, and I'm getting one of my notorious migraines. I take some pills and a quick 10 minute nap in hopes of stopping it, but to no avail. Ahh, to be mortal again. Fortunately the kids are still with my in-laws for the evening, so when I get home, my wife takes one look at me and says "Go to bed." I kiss her gratefully and crash.
Fin.
For those of you that have made it this far. I thank you for taking the time to read this. This has kind of evolved a little larger than I had thought, and it's been kind of nice to be able to extract my though process and interactions on a huge matter such as the infection. I've gotten a lot of positive feedback from it, including the comment "I esp love how you turned something frustrating like a virus infestation into something fun to read about. 😃 Now that's talent! 😃"
We're still on the "front line" as far as other entities being infected with this, and it dawned on me that I should probably throw together a more formal "tech sheet" of what we encountered and links to the tools to remove it. You should see that in the next couple of days.