How can I filter robot crawler hits out of my Apache access_log file?

Dave's Answer:

This is a common problem with log files on Unix and Linux servers, actually. Among the many files that often grown without bounds are the Apache access_log and error_log files and the system /var/log/messages file (on some systems it's called system.log, but it's the same file). If you don't pay attention, these files can quickly grow to be tens, hundreds, or even thousands of megabytes.

Once you have these huge log files that are eating up a significant percentage of your available disk space, your choices are quite limited, as you have learned.

If you have space, the obvious way to weed out the web crawler hits, assuming that you know a unique string that identifies those queries, is to do:

$ grep -v ptrn access_log > new_access_log

However, you don't have space, so here's how I would handle this sticky situation...

First, move the file to a new name:

$ mv access_log bad_access_log

This first step lets you stop the access_log from growing even bigger while you're working on it, then:

$ gzip bad_access_log

creates a compressed '.gz' version of the file that should be about 50% smaller. Now you've probably just freed up about 100MB of space, so you should be able to do something like this:

$ zcat bad_access_log.gz | grep -v ptrn > good_access_log

The zcat command (actually a link to the gzip program, but that's just useless geeky info you can safely ignore) uncompresses the file, but since it feeds the uncompressed result directly to the command pipe, there's no need to reclaim the extra disk space as it processes the data.

If that STILL doesn't work, you could also try:

$ zcat bad_access_log.gz | \
 grep -v ptrn | gzip > good_access_log.gz

Once that's done, remove the bad file and uncompress the good one with:

$ rm bad_access_log*
$ gunzip good*gz

and you should be good to go!