Industry guru Dave Taylor offers tech support on technical and business topics, including iPhone, iPod, Microsoft Windows, Sony PSP, cellphones, online advertising, CSS, Web design, business, Unix, Linux, SEO, Mac OS X, and shell script programming.     


How do I selectively copy files from a directory structure?

Dave, I want to copy a directory tree BUT I want to specify the root of the operation and the destination directory and then specify name/type patterns a la find. The result I want is the root at the destination and then only the directories which have files that match the pattern and the files themselves. Does that make sense?


Dave's Answer:

What you're asking to accomplish here is a bit tricky, because you want to create subdirectories on-the-fly as needed. Not impossible, for sure, but a bit more tricky than a standard find . -name pattern -print or similar.

To make sure I got this one right, I created a simple directory hierarchy of my own for testing purposes, using mkdir and touch. Then I experimented with both the tar and cpio commands...

Before I show you the results, let me say that if you haven't poked around with the cpio command, you should, because it has one great feature that standard tar lacks: it can read a list of filenames from standard input, which makes it an excellent partner to the find command. Some versions of tar can read from stdin, but why it's not a standard part of the command I still don't know. It's not that hard to do...

Anyway, I digress. Here are the two commands I used to create a test heirarchy:

$ mkdir test test2 test/a test/b test/c
$ touch test/a/{x.x,x.y} test/b/x.y test/c/x.x

Running a recursive ls produces this:

$ ls -R test test2
test:
a/      b/      c/

test/a:
x.x     x.y

test/b:
x.y

test/c:
x.x

test2:
$

Now, the challenge you've posed is to be able to copy all the "*.x" files into the new directory, building any necessary subdirectories required as we go. The first solution is to use tar and create an intermediate backup file:

$ cd test
$ tar cvf test.tar $(find . -name '*.x' -print)
./a/x.x
./c/x.x
$ 

This slick command (the command in the $( ) sequence is run, then its output replaces it on the command line before the tar command is actually invoked) generates a file that we can then easily unpack in the "test2" directory with a simple tar xf test.tar. But. But it's not an elegant solution because we don't want to use an intermediate file, plus there's a finite limit to how long a command line can be, and you could easily exceed it here. I know we could use two invocations of tar to copy a directory tree without producing an intermediate archive file, but the command line length limitation would still be lurking.

So instead, here's a much more pleasant solution using cpio instead:

$ find . -name '*.x' -print | cpio -pavd ../test2
/Users/taylor/Desktop/test2/./a/x.x
/Users/taylor/Desktop/test2/./c/x.x
$

And, finally, here's the new heirarchical directory list of "test" and "test2", showing that the latter now does indeed have only the "*.x" files and all necessary subdirectories required too:

$ cd ..
$ ls -R test test2
test:
a/      b/      c/

test/a:
x.x     x.y

test/b:
x.y

test/c:
x.x

test2:
a/      c/

test2/a:
x.x

test2/c:
x.x
$

I bet that'll give you the information you need to accomplish your file copying task, somehow. A fun problem to solve!


Related Unix and Linux Help articles:
✔   Copy and Paste from the Mac OS X Command Line?
I am constantly running commands in Terminal.app on my MacBook and then copying and pasting the results into email messages or documents. Yes,...
✔   Shell script to convert lowercase to title case?
As part of a project I'm working on, I find myself deep in a Linux shell script, needing to have a subroutine that...
✔   Can I script renaming files based on an XML data map?
I have a folder full of files which are named with four digits and a file extension e.g. 0312.file and an XML-file describing...
✔   Test for valid numbers in a Bash shell script?
In a different discussion on this site [see Redirecting input in a shell script] a visitor commented that "I was too busy trying...
✔   Review: iSSH for the iPad/iPhone
If you're running an online business like I am, there are times when you need to connect and log in to the server...

Let's stay in touch!
Sign up for my weekly AskDaveTaylor Newsletter and you'll receive even more tech and gadget help right to your inbox, along with exclusive news and industry updates. It's good stuff. I promise!
    Enter your name: and your email addr:  









Reader Comments To Date: 9

mike said, on January 17, 2007 12:11 PM:

What if you wanted to retain file permissions. I am trying to use this method to copy files modified the last 24 hours, to another directory.

In more detail, I want to two identical directories complete with directory structures. One of the directories I develop in and overnight I want to find any files or new directories and move just those files or directories altered over to the "production" directory of sorts. Does that make sense? I was trying this method but it does not look like proper ownership and permissions are retained.

Dave Taylor said, on January 18, 2007 10:20 AM:

Mike, check out some of the additional flags to the "cp" or "mv" command. On many Unixes the "-p" flag preserves permissions and ownership and you can also do the same sort of thing with a "cpio" invocation, though it's typically a wee bit more complicated. :-)

Scott said, on January 23, 2007 3:01 PM:

This might work for you Mike.
cd /your/source/directory
find . -mtime -1 -depth -print | cpio -pdmuv your/target/directory

mike said, on January 26, 2007 7:16 AM:

That works Scott! Thank you both, that is why Ask Dave Taylor is the best! A knowledgeable host and a great community to back it up.

Joe said, on May 16, 2007 8:08 AM:

Hi Dave,
I am trying to move files from one physical machine to another and still maintain the folder structure in the destination path.
I can create the dir structure using Xcopy [source] [dest] /t but the problem is that I want to ONLY move files that were modified on or before a particular date. So basically if file is older than [DATE], MOVE to [destination path]
Any idea how I can accomplish this please?

Dave Taylor said, on May 17, 2007 6:31 AM:

Honestly, Joe, I would grab the Cygwin Unix toolkit for Windows and download it onto your computer. Then you can use a find|tar sequence to accomplish exactly what you seek quite easily. Start here: http://www.cygwin.com/

skztr said, on December 16, 2008 7:39 AM:

couldn't a combination of --files-from /dev/stdin and piping work with tar?

John L. said, on July 23, 2009 2:08 PM:

for windows, xcopy is more powerful that it looks. If I remember correctly there are undocumented switches you would have to google to find. I ultimately just used the free "XXCOPY" which is more powerful and better documented. Now xcopy has been deprecated by RoboCopy and TeraCopy.

My question relates to your *nix solution. I'm big into making my own directory structures for music or whatnot. Xcopy allowed me to easily copy just an entire directory tree without the files, in effect emptying the tree. In the past I looked into using rsync for this. I think your solution is very close to allowing me to do that elegantly in *nix... Any thoughts?

Jim said, on July 23, 2010 5:16 PM:

Life saver. Thanks. I'm starting to understand why people like linux...

Starbucks coffee cup I do have a lot to say, and questions of my own for that matter, but first I'd like to say thank you, Dave, for all your helpful information by buying you a cup of coffee!

I do have a comment, now that you mention it!











I will never send you any unsolicited email. Ever.






Check This Out Too...

 
Look for Answers
Need Help? Ask Dave Taylor!
Powered By
Linux Journal: Free Issue!


Follow Me on Pinterest

Find Me on Google+
ADT on G+
© 2002 - 2013 by Dave Taylor. All Rights Reserved.

Note: This web site is for the purpose of disseminating information for educational purposes, free of charge, for the benefit of all visitors. We take great care to provide quality information. However, we do not guarantee, and accept no legal liability whatsoever arising from or connected to, the accuracy, reliability, currency or completeness of any material contained on this web site or on any linked site. Further, please note that by submitting a question or comment you're agreeing to my terms of service, which are: you relinquish any subsequent rights of ownership to your material by submitting it on this site. My lawyer says "Thanks".
"Ask Dave Taylor®" is a registered trademark of Intuitive Systems, LLC.