August 8, 2008

Given recent announcements in mass media and numerous questions directed to our partners concerning our decision to abandon the comparative review of anti-virus products by Virus Bulletin we consider it necessary to issue our official statement on this subject.

Virus Bulletin is one of the most respected titles devoted to prevention, detection and removal of malware and spam naturally knitting virtually all anti-virus developers; annual anti-virus conferences held by the magazine still remain a unique event where competitiveness on the market loses its significance as vendors focus on pointing out new trends in the evolution of malware and work out methods to protect users all over the world.

The comparative reviews of anti-viruses conducted by Virus Bulletin every two months is an established event, almost a ritual. Dr.Web is one of the oldest participants with the successful history dating back to the second test in 1998. The testing always stood out among others of its kind for its transparent methods, accuracy and unbiased assessment of products of all vendors and perfect communication with anti-virus companies.

However developments of the industry in last years make many vendors question the comparative reviews. Though transparent and accurate the testing methods fail to keep up with the evolution of malware as well as anti-virus applications. That’s why the long prestigious VB100% can no longer serve as a benchmark reflecting the actual quality of an anti-virus and which is worse is nowadays used to manipulate opinion of users.

Doctor Web sees the issues of the comparative testing as follows:

1. Testing of an anti-virus for VB100% is based on In-the-Wild set of viruses which includes only malware capable of replicating itself which surely narrows the list of malicious programs used for the testing. As estimated by Doctor Web the In-the-Wild collection includes only 10 per cent of the total number of malware modern anti-viruses protect against.
2. The above-mentioned criterion applied to In-the-Wild collection leaves out the large segment of the present-day malware – Trojans. The same applies to one of the gravest IT security issues of last 4-5 years, so called rootkits. No matter how good an anti-virus is at detecting Trojans which outnumber viruses manifold, mo matter what are its rootkit counteraction capabilities it will only get the VB100% upon a successful detection of several thousands of samples from the In-the-Wild collection. Alas, VB100% used as an ultimate benchmark by some marketing specialists and industry experts won’t show a user if an anti-virus is really efficient against Trojans.
3. In order to address new challenges Dr.Web is developing as all other AV products. AV vendors have to deal with new technologies of virus-writers on daily basis which makes constant bringing of innovations into an anti-virus a must. And here regular updates of a virus database are not enough. The testing for VB100% doesn’t compare technical innovations of anti-viruses developed to counteract malicious programs that are never included the In-the-Wild collection.
4. It’s not a routine scan of a collection of files that shows how good an anti-virus is. It is a malicious attack when malware is attempting to get to a computer or a computer has already been infected. Recent years saw numerous proposals to create tougher conditions for testing anti-viruses and assess them by their ability to cope with an active infection. An anti-virus can show astounding results detecting samples from In-the-Wild collection but users will never know if it is the same perfect when malware is running in the RAM and controls the system rather than stored on a hard drive. Neither the test compares curing capabilities of anti-virus products.

Doctor Web considers these issues to have negative impact on the efficiency of the comparative reviews conducted by Virus Bulletin using existing testing methods. Results of the reviews don’t provide any reasonable assessment of the quality of products in question, of their capability to protect users against contemporary threats. The procedure is called the comparative review but in fact the testing by Virus Bulletin doesn’t compare many features implemented in present-day anti-viruses. Under the circumstances VB100% presented as a high-quality mark awarded to a successful participant in truth is merely an evidence of passing a certain test that doesn’t have much to do with trying out capabilities of the software that are really relevant for counteracting contemporary malware threats.

All these issues have led Doctor Web to stepping aside as a participant of the Virus Bulletin comparative reviews. However, we are watching over the evolution of testing methods very closely and are sure to rejoin as soon as they are up to the present day requirements for anti-virus security.