How can I quickly create millions of tiny files in Linux?
Dave, I have searched the web over and can't seem to find an appropriate answer to this question. I have a need for a script that will create millions of 1-5Kbyte files in a linux filesystem for testing purposes. I have to do this on a regular basis and it would be really cool to have it scripted. I am not sure what the quickest way to do this would be?
A bit weird as questions go, but I understand exactly what you're talking about and the good news is that you can accomplish this task quickly and easily without having to have a "for i=1...1000000 do; touch x; done".
The trick is to use a command called split which, as the man page explains, "The split utility reads the given file (or standard input if no file is specified) and breaks it up into files of 1000 lines each."
How would you use this, then? Create a big file and use split with its useful "-b size" flag to create the millions of files you need. Let's step through the process...
Let's assume that you want to have, literally, a million 2.5K files. Now let's assume that you want to run the command one hundred times to produce this, which means that the source file needs to be 1/100th of 2.5 million kilobytes in size, or 250 megabytes. (yeah, I know, a K is 1024 bytes, not 1000 bytes)
Go look at your log files: you might well have a file sitting in /var/log or a similar location that's already that size. I have a /var/log/messages file on my system that clearly needs to be trimmed - it's 381MB in size. Perfect!
You can find your own really big files by using find. Here's how I scrounged my server to find something really big:
$ sudo find . -type f -size +10000 -print
Typically, find works in terms of 512-byte blocks on the size specifier, so this search isn't looking for files that are greater than 100MB, but greater than 50MB. No worries, though, it will invariably find a dozen or more files and one of 'em will be perfect for our needs. If you can't find one that's quite the right size, create the size you need:
$ ls -lh ./private/var/vm/swapfile0 -rw------T 1 root wheel 64M 10 Feb 08:21 ./private/var/vm/swapfile0 $ cat "./private/var/vm/swapfile0" "./private/var/vm/swapfile0" > BigFile $ ls -l /tmp/BigFile -rw-r--r-- 1 taylor wheel 134217728 10 Feb 09:57 /tmp/BigFileNow we have a good size file, one that'll cover your needs. Time to move into a subdirectory and split out oodles of subfiles:
$ ls | head xaa xhe xoi xvm ycq yju yqy yyc zfg zmk xab xhf xoj xvn ycr yjv yqz yyd zfh zml xac xhg xok xvo ycs yjw yra yye zfi zmm xad xhh xol xvp yct yjx yrb yyf zfj zmn xae xhi xom xvq ycu yjy yrc yyg zfk zmo xaf xhj xon xvr ycv yjz yrd yyh zfl zmp xag xhk xoo xvs ycw yka yre yyi zfm zmq xah xhl xop xvt ycx ykb yrf yyj zfn zmr xai xhm xoq xvu ycy ykc yrg yyk zfo zms xaj xhn xor xvv ycz ykd yrh yyl zfp zmtVoila! I think you can run with it from here. Good luck.
More Useful Unix and Linux Help Articles:
✔ Copy and Paste from the Mac OS X Command Line?
I am constantly running commands in Terminal.app on my MacBook and then copying and pasting the results into email messages or documents. Yes,...✔ Shell script to convert lowercase to title case?
As part of a project I'm working on, I find myself deep in a Linux shell script, needing to have a subroutine that...✔ Can I script renaming files based on an XML data map?
I have a folder full of files which are named with four digits and a file extension e.g. 0312.file and an XML-file describing...✔ Test for valid numbers in a Bash shell script?
In a different discussion on this site [see Redirecting input in a shell script] a visitor commented that "I was too busy trying...✔ Review: iSSH for the iPad/iPhone
If you're running an online business like I am, there are times when you need to connect and log in to the server...
Let's stay in touch!
Sign up for my weekly AskDaveTaylor Newsletter and you'll receive even more tech and gadget help right to your inbox, along with exclusive news and industry updates. It's good stuff. I promise!
I do have a comment, now that you mention it!
Check This Out Too...
Look for Answers
All Our Categories
Apple iPad Help
Articles and Reviews
Auctions and Online Shopping
Blogs and Blogging
Building Web Site Traffic
Business and Management
Computer and Internet Basics
d) None of the Above
Google Gmail Help
Google Plus Help
Industry News and Trade Shows
iPhone and Cell Phone Help
iPod, Sony PSP and MP3 Player Help
Kindle Fire Help
Mac OS X Help
Pay Per Click (PPC) Advertising
Search Engine Optimization (SEO)
Shell Script Programming
Tech Support Video Help
The Writing Business
Twitter, LinkedIn and Social Network Help
Unix and Linux Help
Video Game Tips and Help
Windows PC Help
Find Me on Google+
ADT on G+