On a mailing list I’m on, a member recently asked: “Due to an abusive web crawler, I now have a 230MB Apache access_log file on my Web server. I tried to trim it down using grep, but I don’t have enough disk space for the command to succeed. Help!”
This is a common problem with log files on Unix and Linux servers, actually. Among the many files that often grown without bounds are the Apache access_log and error_log files and the system /var/log/messages file (on some systems it’s called system.log, but it’s the same file). If you don’t pay attention, these files can quickly grow to be tens, hundreds, or even thousands of megabytes.
Once you have these huge log files that are eating up a significant percentage of your available disk space, your choices are quite limited, as you have learned.
If you have space, the obvious way to weed out the web crawler hits, assuming that you know a unique string that identifies those queries, is to do:
$ grep -v ptrn access_log > new_access_log
However, you don’t have space, so here’s how I would handle this sticky situation…
First, move the file to a new name:
$ mv access_log bad_access_log
This first step lets you stop the access_log from growing even bigger while you’re working on it, then:
$ gzip bad_access_log
creates a compressed ‘.gz’ version of the file that should be about 50% smaller. Now you’ve probably just freed up about 100MB of space, so you should be able to do something like this:
$ zcat bad_access_log.gz | grep -v ptrn > good_access_log
The zcat command (actually a link to the gzip program, but that’s just useless geeky info you can safely ignore) uncompresses the file, but since it feeds the uncompressed result directly to the command pipe, there’s no need
to reclaim the extra disk space as it processes the data.
If that STILL doesn’t work, you could also try:
$ zcat bad_access_log.gz | \ grep -v ptrn | gzip > good_access_log.gz
Once that’s done, remove the bad file and uncompress the good one with:
$ rm bad_access_log* $ gunzip good*gz
and you should be good to go!
Worth noting is that some Unix and Linux systems have an additional command called zgrep and if you have that, then the sequence of
zcat bad_access_log.gz | grep -v ptrn | gzip > good_access_log.gz
can be simplified to
zgrep -v ptrn bad_access_log.gz | gzip > good_access_log.gz