|
|
How can I quickly create millions of tiny files in Linux?Dave, I have searched the web over and can't seem to find an appropriate answer to this question. I have a need for a script that will create millions of 1-5Kbyte files in a linux filesystem for testing purposes. I have to do this on a regular basis and it would be really cool to have it scripted. I am not sure what the quickest way to do this would be? A bit weird as questions go, but I understand exactly what you're talking about and the good news is that you can accomplish this task quickly and easily without having to have a "for i=1...1000000 do; touch x; done". The trick is to use a command called split which, as the man page explains, "The split utility reads the given file (or standard input if no file is specified) and breaks it up into files of 1000 lines each." How would you use this, then? Create a big file and use split with its useful "-b size" flag to create the millions of files you need. Let's step through the process... Let's assume that you want to have, literally, a million 2.5K files. Now let's assume that you want to run the command one hundred times to produce this, which means that the source file needs to be 1/100th of 2.5 million kilobytes in size, or 250 megabytes. (yeah, I know, a K is 1024 bytes, not 1000 bytes) Go look at your log files: you might well have a file sitting in /var/log or a similar location that's already that size. I have a /var/log/messages file on my system that clearly needs to be trimmed - it's 381MB in size. Perfect! You can find your own really big files by using find. Here's how I scrounged my server to find something really big: $ sudo find . -type f -size +10000 -print Typically, find works in terms of 512-byte blocks on the size specifier, so this search isn't looking for files that are greater than 100MB, but greater than 50MB. No worries, though, it will invariably find a dozen or more files and one of 'em will be perfect for our needs. If you can't find one that's quite the right size, create the size you need: $ ls -lh ./private/var/vm/swapfile0 -rw------T 1 root wheel 64M 10 Feb 08:21 ./private/var/vm/swapfile0 $ cat "./private/var/vm/swapfile0" "./private/var/vm/swapfile0" > BigFile $ ls -l /tmp/BigFile -rw-r--r-- 1 taylor wheel 134217728 10 Feb 09:57 /tmp/BigFileNow we have a good size file, one that'll cover your needs. Time to move into a subdirectory and split out oodles of subfiles: $ ls | head xaa xhe xoi xvm ycq yju yqy yyc zfg zmk xab xhf xoj xvn ycr yjv yqz yyd zfh zml xac xhg xok xvo ycs yjw yra yye zfi zmm xad xhh xol xvp yct yjx yrb yyf zfj zmn xae xhi xom xvq ycu yjy yrc yyg zfk zmo xaf xhj xon xvr ycv yjz yrd yyh zfl zmp xag xhk xoo xvs ycw yka yre yyi zfm zmq xah xhl xop xvt ycx ykb yrf yyj zfn zmr xai xhm xoq xvu ycy ykc yrg yyk zfo zms xaj xhn xor xvv ycz ykd yrh yyl zfp zmtVoila! I think you can run with it from here. Good luck.
More Useful Unix and Linux Help Articles:
✔ Copy and Paste from the Mac OS X Command Line?
I am constantly running commands in Terminal.app on my MacBook and then copying and pasting the results into email messages or documents. Yes,...
✔ Shell script to convert lowercase to title case?As part of a project I'm working on, I find myself deep in a Linux shell script, needing to have a subroutine that...
✔ Can I script renaming files based on an XML data map?I have a folder full of files which are named with four digits and a file extension e.g. 0312.file and an XML-file describing...
✔ Test for valid numbers in a Bash shell script?In a different discussion on this site [see Redirecting input in a shell script] a visitor commented that "I was too busy trying...
✔ Review: iSSH for the iPad/iPhoneIf you're running an online business like I am, there are times when you need to connect and log in to the server...
Let's stay in touch!
Sign up for my weekly AskDaveTaylor Newsletter and you'll receive even more tech and gadget help
right to your inbox, along with exclusive news and industry updates. It's good stuff. I promise!
Categorized:
Unix and Linux Help
(Article 3887,
Written by Dave Taylor)
Tagged: Previous: How come I get lots of spam that isn't addressed to me? Next: How come a URL with "mp3" in it kills Movable Type? Reader Comments To Date: 18Arvind said, on February 11, 2005 3:16 AM:
Hi, With reference to the article about creating many tiny files, if you don't want to search for big files, you can create your own big files using dd, like so: dd if=/dev/zero of=myfile bs=1024 count=1024 This will create a file called "myfile", of size 1MB. You can play around with the block size (bs) and block count (count) options to get the size you need. Then, you can use "split". You can also use "if=/dev/random" to get pseudo-random content in the file. Arvind Dave Taylor said, on February 11, 2005 4:11 AM:
Wally, superb point. One way to see if you have enough inodes is to use the "df" command, of course. If I run "df" on my Mac OS X system, for example, I find out that my main drive has 4,232,612 inodes available and 8,624,197 inodes available, meaning that I'm using 33% of the inodes available. If I want to make a million files then I want at least that many inodes, but on this drive, I have 8.6 million. Plenty. Dave Taylor said, on February 11, 2005 4:13 AM:
Thanks for the tip, Arvind. Somehow I always forget about "dd". And I've been using Unix for 25 years now. :-) Dave Dales said, on February 11, 2005 7:33 AM:
Thanks for the help! I know my question is somewhat strange but it's for a strange project. I have a customer with an expensive SAN system that has literally millions of files. A great number of these files are old and he wants to move the older files to a less expensive storage archive. I have written the script to find the files, build that list of files into a plain text file, then use the list to move and delete the files that were successfully moved. In order to test this, I needed a filesystem with literally millions of files. Your answer was exactly what I was looking for! Thanks again! created a 10M file with DD split -b 5 -a 10 myfile George Rogers Clark said, on February 11, 2005 1:04 PM:
What's with the addition problem? In my humble opinion, a perl script would be the best way to create bunchs of tiny files on Linux. It's been years since I wrote any perl, but I'll just bet some perl guru could show perl to be the one really "right" solution. martin cohen said, on February 15, 2005 12:17 AM:
Since split can get its input from standard input, why not pipe a script into it. As an example, since I use gawk, I might do this: gawk 'BEGIN{for(i=0;i<100000;i++)print i;exit}' | split --lines=10 would create 10000 10-line files. Dave Taylor said, on February 15, 2005 1:32 AM:
Martin, you demonstrate what's so wonderful about Unix: there are always a dozen ways to solve any given problem. An ingenious solution, thanks. carlos said, on March 7, 2005 1:22 AM:
Dear Dave: I read your answer about how to create a million files... I was thinking..(maybe I didn�t understand the question)... first of all.. whats wrong with the first answer you propose? ... second.. whiy not a tiny script that start another... tiny scripts.. (recursively.. I say).. so you can have several shells doing the job.. in the end... interesting question.. just for the fun of it...(I remember when the people working in Basic,.. put this problems and try to resolve them in very few lines...) thanks for your attention.. and your effort in resolve questions.. (and, please, excuse my bad english.. as you can see this is not my maternal language..!!!) good regards, Dave Taylor said, on March 7, 2005 2:40 AM:
Hi Carlos. Your idea of a shell script that recursively calls another shell script could work, but you'd have a lot of processes running, enough that it might well crash your system before you get the million files you seek. k nard said, on April 27, 2005 10:34 PM:
This may be very time-consuming. After you've Jim said, on May 8, 2007 12:13 PM:
Just a clarification, Dave. On Linux, you have to use the -i, or df -i, switch to show inode information. I can't find the equivalent on Solaris 8, 9 or 10. jason said, on June 26, 2007 11:04 AM:
yes | head -c 10m | split -b 1 Will give you 10 million 1 byte files. kjteoh said, on October 5, 2007 9:06 PM:
on Solaris David said, on February 1, 2010 11:16 PM:
An easier way to do this is to use the dd command with a for loop to create many files at the command line. Example below creates a million files each 1 byte each. for((X=0;X<1000000;X+=1)); do dd if=/dev/zero of=file$X.log bs=1 count=1; done Saurabh said, on May 31, 2010 7:03 AM:
Hi Dave, Is there a way by which we can create huge files very quickly. the dd command takes a lot of time. thnx in advance klode said, on May 3, 2011 8:35 PM:
Dave Taylor: Great use of split; I've never thought of using it that way. However no need to find/create an existing big file, just use /dev/zero. To generate 5M 4095-byte files (20,475,000,000 bytes total): Dave Dales: I realize you still might want to use a script if you're doing a bunch of verification, but the original find/copy work can be done with one pipeline. Assuming you want to copy files older than two years: George Rogers Clark: It depends on how you define "best way". Likely the most important issue here is speed: how to generate millions files as quickly as possible. An interpreted, string-handling script language probably isn't the way to go here, but then again I've not done perl speed tests for this scenario. Martin cohen: same concern about the speed of an interpreted, string-handling script language. k nard: Good point, and I'd also test the speed of restoring from a compressed sector-level copy, similar to how Ghost/Acronis/Clonezilla would handle it. jason: Dude, that's just sick :-). But thanks for teaching me about the "-c" option to "head". David: instantiating 1M processes in a for loop will add a serious amount of time. Much better to use a single instance (of whatever command) when handling large numbers of files. This is one of the reasons that xargs was created. Saurabh: if all you want to do is fill memory, don't bother actually creating the file. Instead use dd to read from a drive and discard: Hope this helps, Dave Taylor said, on May 3, 2011 9:39 PM:
Thanks for your extensive comment, klode. The ownership rights issue is on the recommendation of my attorney and is just to ensure that if I ever wanted to use any material from one of my blog posts or a comment (which I would, of course, always attribute properly) that I wouldn't risk subsequently having someone sue me for a copyright violation. No nefarious purpose at all.
I do have a comment, now that you mention it!Check This Out Too... |
Recent Entries
Look for Answers
Recommended
All Our Categories
Apple iPad Help
Articles and Reviews Auctions and Online Shopping Blogs and Blogging Building Web Site Traffic Business and Management Computer and Internet Basics d) None of the Above Facebook Help Google Gmail Help Google Plus Help HTML, JavaScript and Web Site Programming Industry News and Trade Shows iPhone and Cell Phone Help iPod, Sony PSP and MP3 Player Help Kindle Fire Help Mac OS X Help Pay Per Click (PPC) Advertising Pinterest Help Search Engine Optimization (SEO) Shell Script Programming Tech Support Video Help The Writing Business Twitter, LinkedIn and Social Network Help Unix and Linux Help Video Game Tips and Help Windows PC Help Find Me on Google+ ADT on G+ |
but will you have enough inodes?