The reality isn't quite as glamorous. I'm my own lab "staff," and my rows of machines are virtual ones . I can't reasonably test every spyware threat out there, so I rely on a strong, representative sampling. The results are reliable and reproducible, even though I don't have a huge staff of lab-coated assistants.
Spyware samples start to stink after a while: The longer they've been around, the more likely that even the most mediocre antispyware program will handle them (if you've bothered to install one—which isn't always the case, as we've recently shown). So several times a year, I throw them all out and gather a new collection. I start by asking a large group of security vendors to recommend some especially nasty threats, along with where to download each one so I can get them fresh off the Web. Once the vendors have weighed in, I compile their recommendations into one big list and use it to choose a representative selection, making sure to use something from every vendor. Items that are suggested multiple times almost always make the cut.
I also test the products against commercial keyloggers--programs that secretly record your keystrokes and other actions—for a number of reasons. First of all, most antispyware products specifically claim to remove keyloggers, along with spyware, adware, Trojans, and other types of malware. In fact, there were quite a few keyloggers in the list of vendor recommendations for malware samples. These threats are among the most dangerous as far as actual spying goes. They can capture passwords, financial data, or any personal information--instant identity theft! On the other hand, you're less likely to encounter one of these since most can only be installed by someone with physical access to your computer. When analyzing a product's performance, I give commercial keyloggers significantly less weight.
Then I put my virtual machines to work, loading three or four of the selected threats onto each one (since a real-world infested system usually has multiple problems). In theory, testing a security program's spyware removal feature is fairly straightforward. I simply install it on all the test machines and run the deepest scan available. Whatever it asks of me, I do. If it wants to reboot and rescan, even several times, I do it. If it asks to restart in Safe Mode and scan, I do that. If it wants me to fetch an iced latte, off to Starbucks I go (okay, maybe I just do that last part because I feel like it). Then, when the hubbub is over, I check to see if it really did remove the malware—and that's where things get hairy.
Some of the samples are big, blaring, oafish things, plastering my screen with so many overlapping pop-ups that I can hardly find the button to start a scan. Others shoehorn unwanted toolbars into my Web browser, or add desktop icons that won't stay deleted. When that kind of behavior stops, it's pretty clear something is gone. But what about the sneaky stuff that isn't visible? Or the rootkit-based threats that hide so even Windows can't see them? When it comes to making sure the threats are gone, the sleuthing involved is almost fun. Almost.
Tracking spyware requires a bit of prep. First, I boot up a clean virtual machine and launch utilities that monitor changes to the Registry, file system, and active processes. (I'm not naming the utilities, because I don't want to give away all the ingredients in the secret sauce!) Next, I release the sample spyware and let it do its worst. The monitoring utilities produce a flood of information—way more than I can use—so I wrote a custom filter program that discards the obvious dreck and presents the rest in a form that's easy to eyeball. It also lets me cherry-pick the file and Registry traces that are clearly related to the spyware installation and store those in a database. I don't try to record every single trace, just a representative sampling for a spot-check (some spyware apps install dozens of traces). After a spyware removal scan, another handmade program cross-checks that database, alerting me if any of those traces were left behind. If an app leaves a few nonexecutable files behind, I let it slide—as long as there's nothing left that can do anything.
Validating a program's ability to keep spyware from infecting a clean system is easier. I start by installing the app in question on a virtual machine and recording a snapshot of the protected system. Then I revert the VM to that snapshot each time I release a new spyware sample. If there's any doubt that the antispyware blocked it, I can run the spot-check program to verify.
