Industry guru Dave Taylor answers free tech support questions about a wide variety of business and technical topics, including blogging, Google AdSense, MySpace, Sony PSP, Apple iPod, Mp3 players, management, Linux, SEO, Mac OS X, Facebook, Twitter, LinkedIn and Microsoft Windows.

How do I selectively copy files from a directory structure?

Dave, I want to copy a directory tree BUT I want to specify the root of the operation and the destination directory and then specify name/type patterns a la find. The result I want is the root at the destination and then only the directories which have files that match the pattern and the files themselves. Does that make sense?


Dave's Answer:

What you're asking to accomplish here is a bit tricky, because you want to create subdirectories on-the-fly as needed. Not impossible, for sure, but a bit more tricky than a standard find . -name pattern -print or similar.

To make sure I got this one right, I created a simple directory hierarchy of my own for testing purposes, using mkdir and touch. Then I experimented with both the tar and cpio commands...

Before I show you the results, let me say that if you haven't poked around with the cpio command, you should, because it has one great feature that standard tar lacks: it can read a list of filenames from standard input, which makes it an excellent partner to the find command. Some versions of tar can read from stdin, but why it's not a standard part of the command I still don't know. It's not that hard to do...

Anyway, I digress. Here are the two commands I used to create a test heirarchy:

$ mkdir test test2 test/a test/b test/c
$ touch test/a/{x.x,x.y} test/b/x.y test/c/x.x

Running a recursive ls produces this:

$ ls -R test test2
test:
a/      b/      c/

test/a:
x.x     x.y

test/b:
x.y

test/c:
x.x

test2:
$

Now, the challenge you've posed is to be able to copy all the "*.x" files into the new directory, building any necessary subdirectories required as we go. The first solution is to use tar and create an intermediate backup file:

$ cd test
$ tar cvf test.tar $(find . -name '*.x' -print)
./a/x.x
./c/x.x
$ 

This slick command (the command in the $( ) sequence is run, then its output replaces it on the command line before the tar command is actually invoked) generates a file that we can then easily unpack in the "test2" directory with a simple tar xf test.tar. But. But it's not an elegant solution because we don't want to use an intermediate file, plus there's a finite limit to how long a command line can be, and you could easily exceed it here. I know we could use two invocations of tar to copy a directory tree without producing an intermediate archive file, but the command line length limitation would still be lurking.

So instead, here's a much more pleasant solution using cpio instead:

$ find . -name '*.x' -print | cpio -pavd ../test2
/Users/taylor/Desktop/test2/./a/x.x
/Users/taylor/Desktop/test2/./c/x.x
$

And, finally, here's the new heirarchical directory list of "test" and "test2", showing that the latter now does indeed have only the "*.x" files and all necessary subdirectories required too:

$ cd ..
$ ls -R test test2
test:
a/      b/      c/

test/a:
x.x     x.y

test/b:
x.y

test/c:
x.x

test2:
a/      c/

test2/a:
x.x

test2/c:
x.x
$

I bet that'll give you the information you need to accomplish your file copying task, somehow. A fun problem to solve!



Help others find this article at Del.icio.us, Digg, Netscape, Reddit, and Simpy.

Subscribe!

Never miss another useful Q&A article again! Subscribe to AskDaveTaylor with Google Reader.

Comments

What if you wanted to retain file permissions. I am trying to use this method to copy files modified the last 24 hours, to another directory.

In more detail, I want to two identical directories complete with directory structures. One of the directories I develop in and overnight I want to find any files or new directories and move just those files or directories altered over to the "production" directory of sorts. Does that make sense? I was trying this method but it does not look like proper ownership and permissions are retained.

Posted by: mike at January 17, 2007 12:11 PM

Mike, check out some of the additional flags to the "cp" or "mv" command. On many Unixes the "-p" flag preserves permissions and ownership and you can also do the same sort of thing with a "cpio" invocation, though it's typically a wee bit more complicated. :-)

Posted by: Dave Taylor at January 18, 2007 10:20 AM

This might work for you Mike.
cd /your/source/directory
find . -mtime -1 -depth -print | cpio -pdmuv your/target/directory

Posted by: Scott at January 23, 2007 3:01 PM

That works Scott! Thank you both, that is why Ask Dave Taylor is the best! A knowledgeable host and a great community to back it up.

Posted by: mike at January 26, 2007 7:16 AM

Hi Dave,
I am trying to move files from one physical machine to another and still maintain the folder structure in the destination path.
I can create the dir structure using Xcopy [source] [dest] /t but the problem is that I want to ONLY move files that were modified on or before a particular date. So basically if file is older than [DATE], MOVE to [destination path]
Any idea how I can accomplish this please?

Posted by: Joe at May 16, 2007 8:08 AM

Honestly, Joe, I would grab the Cygwin Unix toolkit for Windows and download it onto your computer. Then you can use a find|tar sequence to accomplish exactly what you seek quite easily. Start here: http://www.cygwin.com/

Posted by: Dave Taylor at May 17, 2007 6:31 AM

I have a lot to say, but ...
Starbucks coffee cup I have a lot to say, and questions of my own for that matter, but most of all I'd like to say thank you for all your efforts on this Web site by buying you a chai!

I do have a comment, now that you mention it!









Remember personal info?


Please note that I will never send you any unsolicited commercial email. Ever.

While I'm at it, please note that by submitting a question or comment you're agreeing to my terms of service, which are: you relinquish any subsequent rights of ownership to your material by submitting it on this site.









Search
Find just the answers you seek from among our 1700+ free tech support articles by using our Lijit search engine.


Help!





Subscribe to
Ask Dave Taylor!

Add to Google Reader
Add to My Yahoo!
Subscribe in NewsGator Online

RDF   XML

Free Updates!
Sign up and get free weekly updates and special offers on books, seminars, workshops and more.


Recent Entries
Join the List!
Join my author info mailing list, where you'll learn about my upcoming books, speaking gigs, and more!


Book Links
© 2002 - 2008 by Dave Taylor. All Rights Reserved.

Note: This web site is for the purpose of disseminating information for educational purposes, free of charge, for the benefit of all visitors. We take great care to provide quality information. However, we do not guarantee, and accept no legal liability whatsoever arising from or connected to, the accuracy, reliability, currency or completeness of any material contained on this web site or on any linked site.

[whiteboard marker tray]