How do I search lots of files at once?
Dave, I need to conduct a number of searches through more than 2500 text files. Each search is for a different specific text string. The text files are on my Mac hard drive (Mac OS X 10.3.7) and are arranged into folders within folders within folders. I want the result in a new text file. I think I should be able to do this using some sort of grep or script but cannot figure out how to do it. Please help.
Ah, the serendipity is marvelous! I've just been writing about the find command for the new Tiger edition of my best-selling book Learning Unix for Mac OS X so it's all very fresh in my mind.
Whenever you have a nested file heirarchy that you want to search, you should always, automatically reach for the find monkey wrench, coupled with its partner command xargs. But let's step through this slowly so you can see how these all work together, because we're going to use three different commands in a pipe to accomplish what you seek.
First off, the find command has some of the weirdest syntax in Unix, so if you want to learn more about it, use the man find command within Terminal. For now, just follow along. :-) To find all files below the current point in the file system that are HTML files, you'd use:
$ find . -name "*html" -print
Notice that by not using the pattern *.html this also matches files that have the suffix "shtml" too (typically server-side include HTML). This generates a long list of filenames. To search through them for a specific pattern, you want to use the grep command, as you know, but the wrinkle is that you can't just do something like find | grep because grep just isn't expecting a list of filenames from standard input (stdin).
That's when our pal xargs comes in. The xargs command does expect a list of filenames from stdin and it then acts as a wrapper for Unix commands that don't work that way.
Putting them all together, here's how you could find all HTML files that have a 2004 copyright notice in them, just as a topical example:
$ find . -name "*html" -print | xargs grep 2004 | \ grep '©'
Or, if you want to get fancy, use grep -E '(\© .* 2004)' which is probably better, but it's a bit more complex because it's a regular expression not just a simple pair of patterns. Either way, the result will be a list of filenames and the lines from those files that match.
Now, the last step of your task is to save that output to a new file, which can be done with a simple redirect:
$ find . -name "*html" -print | xargs grep 2004 | \ grep '©' > copyright.2004.txt
I hope that gets you going. Drop that into a shell script then you tweak it to meet the specific pattern and output file naming scheme you need to use.
More Useful Unix and Linux Help Articles:
✔ Copy and Paste from the Mac OS X Command Line?
I am constantly running commands in Terminal.app on my MacBook and then copying and pasting the results into email messages or documents. Yes,...✔ Shell script to convert lowercase to title case?
As part of a project I'm working on, I find myself deep in a Linux shell script, needing to have a subroutine that...✔ Can I script renaming files based on an XML data map?
I have a folder full of files which are named with four digits and a file extension e.g. 0312.file and an XML-file describing...✔ Test for valid numbers in a Bash shell script?
In a different discussion on this site [see Redirecting input in a shell script] a visitor commented that "I was too busy trying...✔ Review: iSSH for the iPad/iPhone
If you're running an online business like I am, there are times when you need to connect and log in to the server...
Let's stay in touch!
Sign up for my weekly AskDaveTaylor Newsletter and you'll receive even more tech and gadget help right to your inbox, along with exclusive news and industry updates. It's good stuff. I promise!
I do have a comment, now that you mention it!
Check This Out Too...
Look for Answers
All Our Categories
Apple iPad Help
Articles and Reviews
Auctions and Online Shopping
Blogs and Blogging
Building Web Site Traffic
Business and Management
Computer and Internet Basics
d) None of the Above
Google Gmail Help
Google Plus Help
Industry News and Trade Shows
iPhone and Cell Phone Help
iPod, Sony PSP and MP3 Player Help
Kindle Fire Help
Mac OS X Help
Pay Per Click (PPC) Advertising
Search Engine Optimization (SEO)
Shell Script Programming
Tech Support Video Help
The Writing Business
Twitter, LinkedIn and Social Network Help
Unix and Linux Help
Video Game Tips and Help
Windows PC Help
Find Me on Google+
ADT on G+