Chapter 2. Operating System Tools

Most of the tools described in this book are application programs. As such, they require an underlying operating system to run on. If you think of these programs as your information security toolkit, then your operating system is your workbench. If your OS is unstable, your security work will suffer; you will never be able to truly trust the data coming from it. In fact, your OS might introduce even more insecurity into your network than you started with. In computer security jargon, having a secure OS to build on is part of what is known as a Trusted Computing Base (TCB). The TCB consists of the entire list of elements that provides security, the operating system, the programs, the network hardware, the physical protections, and even procedures. An important base of that pyramid is the operating system. Without that, you are building your Trusted Computing Base on quicksand.

Chapter Overview Concepts you will learn: Introduction to Trusted Computing Base Guidelines for setting up your security tool system Operating system hardening Basic use of operating system-level tools Tools you will use: Bastille Linux, ping, traceroute, whois, dig, finger, ps, OpenSSH, and Sam Spade for Windows

Many attacks on computers are directed at the operating system. Modern operating systems have ballooned to such size that it is extremely difficult for any one person to completely understand what is going on "under the hood." XP, the most current version of Windows, contains over 50 million lines of code. While it is supposed to be the most secure version of Windows yet (according to Microsoft), new security bugs are found in it almost daily. The more complexity you add to a product, the more likely it is to give unexpected results when given unexpected input. Hackers count on these unexpected results.

It used to be that a computer had a limited number of possible inputs—the application programs that were either designed by or approved by the computer vendor. Now, with the Internet and Java- and Active X-enabled Web browsers, all kinds of traffic and code can come at a computer that the designers never allowed for. The sheer volume of programs combined with the types of traffic coming from the Internet means that operating systems are getting less secure, not more secure, as times goes on, especially when you use them "straight out of the box."

Add to this vendors' tendency to try to make computers as ready as possible so users can simply "plug and play." While some might argue that this is a good thing for the masses of computer illiterates, it is certainly not a good thing from a security standpoint. Most security features are turned off by default, many programs and services are loaded automatically, whether the user will need them or not, and many "extras" are thrown onto the system in an effort to outdo the competition. While Microsoft Windows has been the worst offender in this area, consumer versions of Linux aren't much better, and even server-level operating systems are guilty of this sin. A standard installation of RedHat Linux still loads far too many services and programs than the average user needs or wants. Windows Small Business Server 2000 loads a Web server by default. And while Windows XP improved on the past policy of "everything wide open," there are still insecurities in the product when using the default installation.

Making sure your security tool system is secure is important for several reasons. First of all, if a front-line security device such as a firewall is breached, you could lose the protection that the firewall is supposed to provide. If it's a notification device, for example, an intrusion detection system, then potential intruders could invade the box and shut off your early warning system. Or worse yet, they could alter the data so that records of their activities are not kept. This would give you a false sense of security while allowing the intruders free reign of your network.

There are hacker programs designed to do just this. They alter certain system files so that any data coming out of the machine can be under the control of the hacker. Any computer that has been infected with one of these programs can never be trusted. It is often more cost effective to reformat the drive and start over.

Finally, if unauthorized users commandeer your security box, they could use the very security tools you are using against you and other networks. An Internet-connected machine with these tools loaded could be very valuable to someone intent on mischief.

Ensuring that the base operating system of your security machine is secure is the first thing you should do, before you load any tools or install additional programs. Ideally, you should build your security tool system from scratch, installing a brand new operating system. This way you can be sure that no programs or processes will interfere with your security tools. Also, this guarantees that the base operating system is secure from any previous tampering or malicious programs. If for some reason you have to install your tools on an existing installation of an operating system, make sure you follow the directions later in this chapter for OS hardening and securing your system. Later in this chapter I review Bastille Linux, a tool for doing this on a Linux platform. There are free utilities available from Microsoft for hardening Windows. You can also use the tools described in Chapter 5 to scan an existing system for vulnerabilities.

Your choice of operating system for your security tool system determines how you go about securing it. I recommend an open source operating system such as Linux or BSD, but Windows will work fine as long as you properly secure it first. I used Mandrake Linux to install and run the Linux-based tools recommended in this book, and most Linux distributions and BSD or UNIX operating system can use these tools.

There are many open source operating systems available as mentioned in Chapter 1. Most of them are UNIX-based, although they all have a graphical interface available called X-Windows, and window managers such as KDE and GNOME. These interfaces will be familiar to anyone who has used Microsoft Windows, but there are a few differences.

I do not advocate that one operating system is intrinsically better than the others as far as security goes. It is all in the way you use it and configure it; hence the lengthy section that follows on hardening the OS installation. I used Linux because it is the one I have the most experience with, and I felt that it was compatible with most systems being used. With over 50 million users worldwide and dozens of variants, Linux has the widest variety of programs, and most of the open source security tools I mention in this book are designed specifically for it.

The first tool discussed automates locking down a Linux system. This will ensure you are working with a workstation that is as secure as it can be initially. There are also some basic tips on how to properly secure the Windows operating system for use as a security workstation. Finally, you will use some tools at the operating system level. There are certain system-level functions that you will use regularly in your security applications, and several of these are included in the tools section.

This chapter is not intended to be a definitive guide on securing any of these operating systems, but it gives you an overview of the basics and some tools to use.