
How can I filter robot crawler hits out of my Apache access_log file?
On a mailing list I'm on, a member recently asked: "Due to an abusive web crawler, I now have a 230MB Apache access_log file on my Web server. I tried to trim it down using grep, but I don't have enough disk space for the command to succeed. Help!"
This is a common problem with log files on Unix and Linux servers, actually. Among the many files that often grown without bounds are the Apache access_log and error_log files and the system /var/log/messages file (on some systems it's called system.log, but it's the same file). If you don't pay attention, these files can quickly grow to be tens, hundreds, or even thousands of megabytes. Once you have these huge log files that are eating up a significant percentage of your available disk space, your choices are quite limited, as you have learned. If you have space, the obvious way to weed out the web crawler hits, assuming that you know a unique string that identifies those queries, is to do: $ grep -v ptrn access_log > new_access_logHowever, you don't have space, so here's how I would handle this sticky situation... First, move the file to a new name: $ mv access_log bad_access_logThis first step lets you stop the access_log from growing even bigger while you're working on it, then: $ gzip bad_access_logcreates a compressed '.gz' version of the file that should be about 50% smaller. Now you've probably just freed up about 100MB of space, so you should be able to do something like this: $ zcat bad_access_log.gz | grep -v ptrn > good_access_logThe zcat command (actually a link to the gzip program, but that's just useless geeky info you can safely ignore) uncompresses the file, but since it feeds the uncompressed result directly to the command pipe, there's no need to reclaim the extra disk space as it processes the data. If that STILL doesn't work, you could also try: $ zcat bad_access_log.gz | \ grep -v ptrn | gzip > good_access_log.gzOnce that's done, remove the bad file and uncompress the good one with: $ rm bad_access_log* $ gunzip good*gzand you should be good to go!
Help others find this article at Del.icio.us, Digg, Netscape, Reddit, and Simpy.
Categorized:
Unix and Linux Help
(Article 3791)
Tagged: Previous: What one feature of Apple's Safari browser do you prefer over Firefox? Next: How do I install Lexar's JumpDrive Secure on my Mac? Subscribe!
Never miss another useful Q&A article again! Subscribe to AskDaveTaylor with Google Reader. Worth noting is that some Unix and Linux systems have an additional command called zgrep and if you have that, then the sequence of zcat bad_access_log.gz | grep -v ptrn | gzip > good_access_log.gz can be simplified to zgrep -v ptrn bad_access_log.gz | gzip > good_access_log.gz Posted by: Dave Taylor at December 4, 2004 1:55 PMI have a lot to say, but ...
I do have a comment, now that you mention it!
|
Search
Find just the answers you seek from among our 1700+ free tech support articles by using our Lijit search engine.
Help!
Subscribe to
Ask Dave Taylor!
Free Updates!
Sign up and get free weekly updates and special offers on books, seminars, workshops and more.
Articles and Reviews
Auctions and Online Shopping Blogs and RSS Feeds Building Web site traffic Business and Management Cell Phones and Mobile Phones CGI Scripts and Web Site Programming Computer and Internet Basics d) None of the Above HTML and CSS Mac OS X Help MySpace, Facebook, Twitter and Social Network Help Pay Per Click (PPC) Search Engine Optimization Shell Script Programming Sony PSP, MP3 Players, Etc. The Writing Business Unix and Linux Help Video Game Tips and Help Windows Help
Recent Entries
Join the List!
Book Links
|