Security consultants and value-added resellers (VARs) should look at a variety of alternative antivirus techniques, ranging from the low-tech filtering of attachments to more innovative in-line and heuristic-based antivirus systems in order to keep ahead of the most recent malware threats. This tip, courtesy of SearchSecurity.com's Security School series, will shed light on the latest techniques.
Viruses, worms and Trojan horses represent a constant threat to enterprise networks. As a class, malware has become not only one of the greatest risks to network security, but also one of the areas best covered by existing defenses. While the debates over the usefulness and appropriateness of technologies such as intrusion prevention and content filtering continue, no one seems to disagree with the premise that every network needs a strong antivirus strategy.
Because the threat level and risks of malware are high, enterprises and software vendors have sought additional tools beyond traditional virus scanners to mitigate risks.
The most promising new antimalware technologies are reputation-based systems, both for spam and viruses. As a perimeter tool, reputation-based defense systems are quickly becoming the norm in a wide variety of products, from the IP layer up to the application layer. However, network managers and consultants should look at a variety of alternative techniques, ranging from the low-tech filtering of attachments to more innovative in-line and heuristic-based antivirus systems.
Antivirus is one of the oldest technologies we have for perimeter intrusion defense. At the edge of the network, classic antivirus technology is usually incorporated into dedicated email security appliances designed to scan incoming (and sometimes outgoing) email for malware. That's an active and full market, with literally over a hundred products available to act as email security devices.
When an unchecked malware infection occurs, the risk to a network can be enormous. Although we haven't seen any disastrous viruses lately, many security managers are still very aware of the havoc that worms such as Code Red, SoBig and Sasser caused.
As a further precaution, some enterprises also perform in-line antivirus scanning. In-line antivirus scanners, typically incorporated into firewalls, look at not just incoming and outgoing SMTP traffic, but also other mail protocols (POP and IMAP), Web traffic (HTTP) and often file transfers using FTP. While in-line antivirus scanners are not as flexible or as reliable as standalone antivirus, they can catch a large percentage of virus traffic and can be a valuable adjunct to both desktop and server-based antivirus deployments.
The danger with in-line antivirus is that it's much more difficult to perform antiviral checking on packets flowing across a wire than it is on a file sitting on the end-user's disk. For example, if an end-user uses an SSL connection to read their Yahoo email account, the packets are encrypted -- and the inline virus scanner can't see them. Even if they don't use an encrypted connection, how Webmail services deliver attachments varies from service to service, and an antivirus scanner must be able to reconstruct the file based on what the user sees. That can mean special application-specific intelligence for popular Webmail services and a risk that tomorrow's new Webmail service or gateway won't be properly scanned.
In-line scanners are also hampered because they can only scan traffic they understand, in protocols they expect. While you can often assume that Web traffic will run over TCP port 80 and SMTP over TCP port 25, those assumptions are just unreliable best guesses, especially if the user or software is trying a more crafty approach. An in-line antivirus scanner could, in theory, be looking at all traffic over all ports. However, performance issues related to this approach make it untenable for all but the slowest and smallest of networks. Most Unified Threat Management (UTM)-based antivirus products won't even offer this option because of the performance costs. A simple check box that says "scan all ports" and takes the firewall down to its knees is a dangerous option.
Most in-line network-based intrusion prevention system (IPS) products include some antivirus capabilities. No IPS advertises itself as a full antivirus solution, but some of the most egregious malware can be detected by an IPS.
IPSes are much better at catching malware after it has already infected your network, during propagation. Almost any malware that actively propagates can be detected by its behavior. For example, a malware-infected PC might suddenly start making random connections to mail servers (rather than using the corporate mail server). Or it might try to connect to hundreds or thousands of file servers across the corporate network. This type of propagation can be easily detected by most IPS technologies, and you'll find that IPSes have the capability to block further malware propagation, based on behavior anomalies alone.
If you are looking into in-line antimalware technology, here are some guidelines that will help you choose and configure the best solution:
- Because in-line antimalware does not have total control over the client/server communication that it's intercepting, the notification that a policy violation has occurred can be clumsy. For each of the paths that you plan to intercept, be sure that you understand exactly what an end-user is going to see and how the Web or mail client will respond when inline antimalware has triggered. This knowledge will be an important part of your security training for end users, as well as for the help desk, in solving "strange behaviors" with client systems.
- Inline antimalware can check both incoming and outgoing traffic. While it's great to prevent infections, it's almost as good to know who is infected. Make sure you enable outgoing traffic checking, and create an alerting system so that you know who has become infected. Actively responding to those alerts will help to contain infections and assist end users in understanding what action they might have taken that caused the problem.
Going low tech for email
While malware doesn't exclusively enter via email, the use of email as a preferred infection vector means that email controls are a primary perimeter defense against incoming malware.
The problem with antivirus scanners in email is that there is always a lag between the time a new virus is created and the time when an antivirus scanner has a signature that can catch the virus. Of course, many viral variants and polymorphic viruses can be caught by existing signatures, but there will always be new examples that require signatures or, worse, changes in the scanner to catch them.
Finding malware before a definitive signature is available is one of the goals of any good security program, and network managers have turned to a variety of techniques to help them reach that goal.
One low-tech but surprisingly effective technique has been to simply block many types of attachments (technically MIME body parts) in email messages. The logic is pretty simple: most people in organizations are sharing a small subset of the world of file types, typically Microsoft Office documents, text files and Adobe PDF formatted files. Rather than allow anyone in the organization to get anything from the outside world, network managers have set filtering policies to either only allow specific types of attachments or to block some of the types that are well-known as infection vectors (such as executable files or batch files).
It's a somewhat draconian policy, but it also turns out to be an effective one for protecting against many of the existing viral threats, especially in the enterprise environment. Personally, I don't care for this strategy. I think that we buy virus scanners and other antimalware tools to do this for us and picking up a blunt stick to bludgeon email into submission has higher costs than benefits. However, every environment is different and there are network managers who have evaluated their environment and decided that this is the best strategy. If you choose (or already have chosen) a policy like this, four guidelines will help you arrive at the best solution:
- Email attachments typically have three labels on them: a filename extension, a MIME type and a real file fingerprint, determined by looking at the file and checking the actual file type. Your filtering solution should act on all three of these, not just one. If it turns out that an executable simply has to be renamed to .XLS to get through your filter, then people will discover that in a few minutes and you'll have reduced total effectiveness.
- If you're going to have a policy like this, your best strategy is to have a deny-by-default approach with certain file types allowed. Otherwise, you'll be forever updating the list of "bad" attachment types as new viruses figure out a way to get through common defenses. Once you come up with a reasonable set of file types for your user community, advertise what is and what is not permitted so that this is not a surprise to end users. Don't forget that people are increasingly using archiving tools, such as ZIP, to package up multiple files. You will need to allow at least one of these (but probably not every variation).
- Understand that end users will find ways to work around this policy. The availability of tools like YouSendIt.com (a Web site that allows anyone to upload documents transparently for later download) means that you won't be able to keep every file type out. Since users will always find valid exceptions to this kind of control, you need to both educate them on why you're doing this and how they can safely work around your controls. You also need to be able to make exceptions to the policy so that people can get their work done. If the marketing department is paying someone to generate a new logo in Adobe Illustrator, you need to have a quick way to let them conveniently move that file type around.
- While most malware should be eradicated and the carrier (such as an email message) deleted without notice, this isn't true of deleted attachments. Anytime you take anything out of an email message, you should put something back in to indicate the action you've taken. Obviously, this means that any attachment filtering should happen after normal virus scanning.
Alternative antivirus approaches
Two new approaches have been added to the security manager's arsenal in the fight against malware: heuristic-based and reputation-based antivirus. Some of the ideas behind these technologies are fairly old, but products are now widely available that implement these new strategies.
Heuristic-based antivirus is a fairly old idea, but one that has not gained a strong following. The idea is that viruses can be identified not just by their signature, but by heuristics that look for unusual patterns in certain types of files for example, or by executing the virus in a virtual machine "jail" to examine its behavior. Many antivirus vendors include some heuristic technologies in their scanning engines, but no one has gone out on a limb and suggested replacing signature-based antivirus with heuristic-based systems. Heuristic-based antivirus has more variable and unpredictable false-positive and false-negative rates than normal antivirus software, because the rules used to identify malware are fuzzier and approximate than with a signature-based product.
From a network manager or consultant's point of view, the appearance of a heuristic-based technology is a positive sign that an antivirus vendor is looking for more than just signatures in searching for viruses. However, it's unlikely that heuristic-based antivirus will ever push aside signature-based technologies.
Reputation-based antivirus uses the context of the virus to help decide whether an attachment or connection is likely to have undesirable consequences. Like heuristic-based antivirus, reputation-based systems are an adjunct to virus scanners. Vendors are promoting them for "near zero-day" protection, because they can be used to identify suspect email before an antivirus signature is prepared.
Reputation-based services vary widely in their actual implementation, but have some common elements, including a large centralized reputation database and a voracious appetite for transaction data about the behavior of systems across the Internet. For example, a reputation-based service might collect information about spam and virus senders from its customers and other spam traps. When a system or network is identified as being a source for malware or spam, future messages can be held up or specially tagged because of the poor reputation of the sender.
This can be taken a step further: if you actually accept the suspect messages from suspect senders, you can analyze what they're sending and use that to characterize likely spam or malware from other senders. Reputation-based services use this technique and others, to help identify viruses before the antivirus signatures can.
If new techniques like these interest you, here are some guidelines that will help you choose and configure the best solution:
- There is no magic in this business. When looking at new technologies, a vendor should be able to explain what they are doing and how this improves on standard signature-based antivirus. Any pitch with excessive hocus-pocus or claims of revolutionary advances is likely accompanied by an ineffective product that won't stay the distance.
- Reputation-based technology is one of the most effective ways to knock the top third of "bad traffic" off your network. As long as spammers and malware writers use zombie PCs, this kind of technology will be very valuable. Anyone looking to increase system performance and gain an edge on the malware of tomorrow should be looking carefully at reputation-based services.
About the author
Joel Snyder is a senior partner with Opus One, a consulting firm in Tucson, Arizona. He spends most of his time helping people build larger, faster, safer and more reliable networks. He is a frequent contributor to Information Security magazine and has advised and trained thousands of people privately and at conferences around the world on networking, security, messaging and VPNs.
This tip originally appeared in SearchSecurity.com's Security School series.