Recipe 22.15. Blocking Obnoxious Visitors

22.15.1 Problem

You're getting pummeled by annoying site-scoopers, or search-engine bots, or other troublemakers who are blowing your bandwidth allowance right out of the water. How can you block them from accessing your site?

22.15.2 Solution

Use Apache's access controls in httpd.conf. Use this method to deny specific IP addresses or hosts:

Order allow,deny

Allow from all

Deny from 12.34.56.78

# this one keeps linking to our graphic images

Deny from *.booger.baddomain.net

# darned foo.com searchbot has been hammering us

Deny from search.foo.com

22.15.3 Discussion

When reviewing your logs, it's often an unpleasant surprise to see how much traffic is search-engine bots, or people grabbing your entire site with wget or other web site suckers. A lot of search-engine bots ignore robots.txt and crawl web sites as often and as thoroughly as they like. Automated tools are wonderful, but too many people turn them loose on the Internet with little thought for the consequences.

To learn how to use Webalizer, a tool that will make your log analysis easier so that you can spot these troublemakers quickly, see Recipe Recipe 22.21.

22.15.4 See Also

• http://localhost/manual/howto/auth.html