Chapter 7. Intrusion Detection Systems

In the last chapter you saw the power of a network sniffer and all of the useful things you can do with one. You can even use a sniffer to look for suspicious activities on your network. You can take this a step further with a type of software called an intrusion detection system (IDS). These programs are basically modified sniffers that see all the traffic on the network and actually try to sense potential bad network traffic and alert you when it appears. The primary way they do this is by examining the traffic coming through and trying to match it with a database of known bad activity, called signatures. This use of signatures is very similar to the way anti-virus programs work. Most types of attacks have a very distinctive look at the TCP/IP level. An IDS can define attacks based on the IP addresses, port numbers, content, and any number of criteria. There is another way of doing intrusion detection on a system level by checking the integrity of key files and making sure no changes are made to those files. And there are emerging technologies that merge the concept of intrusion detection and a firewall or take further action beyond mere detection (see the sidebar on "A New Breed of Intrusion Detection Systems"). However, in this chapter I focus on the two most popular ways to set up intrusion detection on your network and systems: network intrusion detection and file integrity checking.

Chapter Overview Concepts you will learn: Types of intrusion detection systems Signatures for network intrusion detection systems False positives in network intrusion detection systems Proper intrusion detection system placement Tuning an intrusion detection system File integrity checking Tools you will use: Snort, Snort Webmin module, Snort for Windows, and Tripwire

A Network Intrusion Detection System (NIDS) can protect you from attacks that make it through your firewall onto your internal LAN. Firewalls can be misconfigured, allowing undesired traffic into your network. Even when operating correctly, firewalls usually leave in some application traffic that could be dangerous. Ports are often forwarded from the firewall to internal servers with traffic intended for a mail server or other public server. An NIDS can watch for this traffic and flag potentially dangerous packets. A properly configured NIDS can double-check your firewall rules and give you additional protection for your application servers.

While they are useful for protecting against outside attacks, one of the biggest benefits of an NIDS is to ferret out attacks and suspicious activity from internal sources. A firewall will protect you from many external attacks. However, once an attacker is on the local network, a firewall does you very little good. It only sees traffic traversing through it from the outside. Firewalls are mostly blind to activity on the local LAN. Think of an NIDS and firewall as complementary security devices, the strong door lock and alarm system of network security. One protects your perimeter; the other protects your interior (see Figure 7.1).

There is good reason to keep a close eye on your internal network traffic. FBI statistics show that over 70 percent of computer crime incidents come from an internal source. As much as we would like to think that our fellow employees wouldn't do anything to hurt us, this is sometimes not the case. Internal perpetrators aren't always moonlighting hackers. They can range from a disgruntled system administrator to a careless employee. The simple act of downloading a file or opening an e-mail attachment can load a Trojan horse that will create a hole in your firewall for all kinds of mischief. With an NIDS, you can catch this kind of activity as well as other computer shenanigans as they happen. A well-tuned NIDS can be the electronic "alarm system" for your network.

A New Breed of Intrusion Detection Systems Anomalous Activity-Based IDS Rather than using static signatures, which can only catch bad activity when it can be explicitly defined, these next-generation systems keep track of what normal levels are for different kinds of activity on your network. If it sees a sudden surge in FTP traffic, it will alert you to this. The problem with these kinds of systems is that they are very prone to false positives. False positives occur when an alert goes off, but the activity it is flagging is normal or allowed for your LAN. A person downloading a particularly large file would set off the alarm in the previous example. Also, it takes time for an anomalous detection IDS to develop an accurate model of the network. Early on, the system generates so many alerts as to be almost useless. Additionally, these types of intrusion detection systems can be fooled by someone who knows your network well. If hackers are sufficiently stealthy and use protocols that are already in high use on your LAN, then they won't set off this kind of system. However, one big upside of this kind of system is that you don't have to continually download signature updates. As this technology matures and becomes more intelligent, this will probably become a popular way to detect intrusions. Intrusion Prevention Systems A new type of NIDS called an Intrusion Prevention System (IPS) is being trumpeted as the solution to enterprise security concerns. The concept behind these products is that they will take action upon alerts as they are generated. This can be either by working with a firewall or router to write custom rules on the fly, blocking activity from suspicious IP addresses, or actually interrogating or even counterattacking the offending systems. While this new technology is constantly evolving and improving, it's a long way from providing the analysis and judgment of a human being. The fact remains that any system that is 100 percent dependant on a machine and software can always be outwitted by a dedicated human (although certain defeated chess grandmasters might beg to differ). An open source example of an IPS is Inline Snort by Jed Haile, a free module for the Snort NIDS discussed in this chapter.