The How and Why of Intrusion Detection and Prevention

By Ray Stirbei

In spite of near ubiquitous implementation of firewalls, security problems persist for organizations of all sizes. Preventive measures such as firewalls sometimes fail to avert security incidents, and hence, a complete security policy should encompass detective and corrective measures as well.
This article presents an overview of intrusion detection and prevention systems and their role in securing networks. These systems are fairly complex. Inherently biased and elusive vendor literature does not make it easier to properly understand and implement this technology.

Intrusion Detection Systems

Intrusion detection systems (IDS) can be introduced in a network as a detective mechanism. Their objective is to identify activities that violate an organization’s security policy. Analogous to a burglar alarm, these systems sound an alert if an attacker has managed to breach the security perimeter. The first intrusion detection systems were developed by the US Department of Defense in the late ’80s, and since then, this technology has been the recipient of a good amount of research and development.

How are intrusions detected? 
Detecting intrusions is an inherently difficult computer science problem. How is malicious activity different from authorized activity and how do we codify this knowledge in a way computers can understand? The general solutions to this problem can be categorized as misuse detection and anomaly detection. Misuse detection attempts to identify activity known to be unwanted, and it relies on attack signatures to do so. A simple signature can be “GET /cg4i-bin/ nph-test-cgi? HTTP/1.0.” This approach is also used by anti-virus scanners.
Pros:

• Good accuracy because signatures are very specific.

Cons:

• Requires constantly updated signatures and generally costly to manage and maintain.
• Does not identify unknown attacks for which it doesn’t have signatures.

This approach tends to have fewer false positives (false alarms of legit activity) but more false negatives (actual attacks missed).

Anomaly detection is the converse of attack signatures. Initially it defines a model of acceptable system activity and attempts to identify activity that doesn’t fit the model. The model is usually a set of behavioral rules. For example, if all users log out at 6:00 P.M. every weekday, and if the system finds a user logged in at 2:00 A.M. on a Sunday, it will flag the anomaly. It is generally very difficult to model acceptable behavior.
It is also difficult to adapt to abrupt changes in normal usage, which can happen frequently in large networks.

Pros:

• Less overhead than the misuse detection approach.
• Able to detect unknown attacks.

Cons:

• Low accuracy; sometimes it cannot detect known attacks.

This approach tends to have more false positives and fewer false negatives. Good anomaly detection comes at the price of persistent false positives.
Both approaches are pattern matching, that is to say, they look for patterns that match some model. Vendors typically refer to pattern matching solely as misuse detection. There are no widely accepted definitions. So watch out for semantics!
There are also many variations of these two detection approaches, such as protocol analysis.
Protocol analysis is similar to misuse detection, but it is less specific and its focus is the protocol header, rather than the payload. It might also do some calculations in order to detect protocol violations. For example, it examines a community string within an SNMP packet and then fires an alert if it is longer than 200 bytes (longer than specified by RFC). Such would be an example of a buffer overflow in UCD-SNMP.
A real-world IDS uses multiple detection mechanisms because different approaches are used to detect different attacks. For example, signatures (misuse detection) readily identify macro viruses (like “I Love You”), cgi exploits, and specific attacks. Anomaly detection is adept at detecting denial of service attacks where one host is establishing many sessions in an extremely short amount of time and tearing the connections down prior to any data exchange.
Other attacks like DNS cache poisoning can only be detected using protocol analysis because it involves comparing DNS requests and replies.

How does an IDS work?
An IDS is comprised of one or more sensors employing one or more detection mechanisms described above. An intrusion sensor can monitor a host or network traffic. Thus we have:

• host-based intrusion detection systems (HIDS) 
• network-based intrusion detection systems (NIDS)

Hybrid IDS systems use a combination of both using centralized event management. Most commercial products are of this type.

HIDS have a good view of what is happening, but they are highly susceptible to attack.

Host-based software attempts to detect by monitoring file integrity or audit log files. Misuse detection takes forms of log file monitors, and anomaly detection is implemented by integrity monitors. Log file monitors parse the system logs for suspicious events matching a set of signatures.

Integrity monitor detects changes (file additions, deletions, modifications) of critical system files by taking a snapshot of your system in a secure known state and continually checking against this baseline. Recent HIDS are also able to perform process monitoring.

NIDS are passive devices resistant to attack, but they do not have a good view of what is happening and thus it is easier to evade them. Sensors on the network work by using a sniffer to capture packets and an

engine to analyze packets and compare them against a database of attack signatures. These signature take the form of text rules or filters based on simple scripting.

Perfection is rarely achieved in computer systems, and IDS suffer from a number of problems, some of them fundamental. Tactical problems of NIDS are large amounts of alerts and false positives, an inability to inspect

encrypted traffic, and being able to track an increasingly amount of network traffic. Evasion, however, is an inherent problem of NIDS. It happens due to the differences in TCP/IP stacks and NIDS protocol implementation, among other technical things. It is significant because IDS evasion techniques are built in

scanning programs (nessus, whisker, nmap) and are available in hacker tools (ADMutate).

HIDS tend to be difficult to deploy ubiquitously and the fundamental problem is running on a potentially compromised host. How can you trust software running inside an untrusted computer?

How to select an IDS

There have been formal IDS evaluations by MIT, DARPA, industry publications (Network World), and testing organizations (Neohapsis OSEC, Miercom, NSS). There is even help5 running such an evaluation yourself.

Like any system, an IDS can be evaluated in numerous ways. Any security measure must be considered within the context of an individual system. The efficacy of an IDS depends on many variables: network topology, size, capacity, traffic streams, etc. As with most things, there is no single IDS that fits every scenario.

Your basic choices are:

1. Commercial product (COTS).
2. Custom solution using open source software (for example, Snort, Argus, SHADOW).
3. Outsource it to a managed security service provider (MSSP).

Choice two is addressed by a SANS paper7 and a Gartner report1 helps with choice three. When you are choosing any software product, you are selecting a product as well as a partner who will be updating that product. A trusted vendor with a good track record is especially critical for an IDS because signatures are time sensitive. Besides the general criteria like the vendor’s financial health and product cost, some

other things to consider are:

• Depth and accuracy of coverage. What is the size and quality of signature database? 
• General architecture. Does it scale? Is it robust? 
• Management and operations. Is it easy to use? 
• Timely updates. What is the lag time of signatures from vulnerability disclosure? 
• Extensible. Does it allow custom signatures? 
• Skill set requirements. How much training does it require?

Network World has tested current vendor solutions on a production network with interesting results3. SC Magazine has also published a comprehensive product review8 recently.
Intrusion Prevention Systems

How does an IPS prevent an attack? In general, an IPS can react in different ways:

• drop the traffic 
• shun (block traffic from an attacker’s host) 
• rate limit 
• terminate process

Traditional IDS products incorporate reactive features like terminating TCP connections that it deems malicious or by updating firewall rules to drop traffic from the host altogether.
However, these reactive features are not widely used due to false alarms and because shunning exposes a network to denial of service attacks or further attacks on the IDS itself. IDS vendors have been adding prevention features to their product lines. ISS RealSecure, for example, has an optional inline component that can actively filter network traffic.

How does an IPS work?
Neil Desai wrote an informative article4 describing the various types of IPS. He categorized the vendors into

• in-line network intrusion detection systems. 
• layer 7 switches. 
• applications firewalls/IDS. 
• hybrid switches. 
• deceptive applications.

IPS is a recent technology. There is debate about whether it can deliver the promise of prevention. Cisco’s purchase of Okena and Network Associates’ purchase of IPS vendors Entercept Security and Intruvert Network may change the dynamics of this market.
How to select an IPS

There are many types of IPS. Most criteria rom IDS selection apply here as well, with the addition of

• integration with your vulnerability assessment tools. 
• availability. Does it fail safe? How does it resist attacks on itself? 
• granularity in administration and automated response.

The SecurityFocus article includes a list of vendors and their type of IPS solution. Many vendors sell appliances, but there are a few open source options to elements that can be done in software: inline protection can be addressed by Snort, application protection by systrace or OWASP CodeSeeker, and deceptive applications by honeypots.

Summary—Why Does IDS/IPS Matter?

Intrusion prevention systems (IPS) by definition take an active role in prevention of an attack. IPS is based on the well-understood parts of IDS. It does not rely on signatures but just like an IDS, it can run on separate hardware or inside hosts.
Why should you use an IDS if it’s so costly and error prone? An IDS can be used to better manage risks in your environment and increase security controls. It will help you have a better understanding of what is happening on your network and its specific threats. It can also provide forensic analysis in case of an intrusion.
Why should you use IPS if it can stop a customer from doing business with your organization?
IPS belongs to a higher risk/higher reward product category. It purchases additional protection and can lessen the urgency to patch, but an incorrect response has ill effects on your network. However, an IPS implementation should very carefully considered and even more so tested!
An IDS and/or IPS improves an organization’s security posture but only if properly understood, implemented and maintained. You can refer to SANS IDS FAQ2 for ideas making the business case for an IDS. Also consider the legal aspects of intrusion detection, delineated here,6 prior to an implementation.
As with any security measure, consider IDS/IPS in the context of your risk model, needs and people. If you don’t have trained engineers to monitor and tune these systems, your return on investment (ROI) will be significantly degraded. Remember that logging events without inspection does not an IDS make.
If you are interested in learning about the forefront of ID research, read papers from RAID conferences. Stephen Northcutt’s book, Network Intrusion Detection, is highly recommended if you want to learn the details of intrusion detection.
Academics say IDS lacks completeness, that is, it does not detect all classes of intrusions another way of saying no single security measure is a panacea, and our best strategy is still defense in depth.