How can I quickly create millions of tiny files in Linux?

but will you have enough inodes?

Hi,

With reference to the article about creating many tiny files, if you don't want to search for big files, you can create your own big files using dd, like so:

dd if=/dev/zero of=myfile bs=1024 count=1024

This will create a file called "myfile", of size 1MB. You can play around with the block size (bs) and block count (count) options to get the size you need. Then, you can use "split". You can also use "if=/dev/random" to get pseudo-random content in the file.

Arvind

Wally, superb point. One way to see if you have enough inodes is to use the "df" command, of course. If I run "df" on my Mac OS X system, for example, I find out that my main drive has 4,232,612 inodes available and 8,624,197 inodes available, meaning that I'm using 33% of the inodes available. If I want to make a million files then I want at least that many inodes, but on this drive, I have 8.6 million. Plenty.

Thanks for the tip, Arvind. Somehow I always forget about "dd". And I've been using Unix for 25 years now. :-)

Thanks for the help! I know my question is somewhat strange but it's for a strange project. I have a customer with an expensive SAN system that has literally millions of files. A great number of these files are old and he wants to move the older files to a less expensive storage archive. I have written the script to find the files, build that list of files into a plain text file, then use the list to move and delete the files that were successfully moved. In order to test this, I needed a filesystem with literally millions of files.

Your answer was exactly what I was looking for! Thanks again!

created a 10M file with DD

split -b 5 -a 10 myfile

What's with the addition problem?

In my humble opinion, a perl script would be the best way to create bunchs of tiny files on Linux. It's been years since I wrote any perl, but I'll just bet some perl guru could show perl to be the one really "right" solution.

Since split can get its input from standard input, why not pipe a script into it.

As an example, since I use gawk, I might do this:

gawk 'BEGIN{for(i=0;i<100000;i++)print i;exit}' | split --lines=10

would create 10000 10-line files.

Martin, you demonstrate what's so wonderful about Unix: there are always a dozen ways to solve any given problem. An ingenious solution, thanks.

Dear Dave:

I read your answer about how to create a million files...

I was thinking..(maybe I didn´t understand the question)... first of all.. whats wrong with the first answer you propose? ... second.. whiy not a tiny script that start another... tiny scripts.. (recursively.. I say).. so you can have several shells doing the job..

in the end... interesting question.. just for the fun of it...(I remember when the people working in Basic,.. put this problems and try to resolve them in very few lines...)

thanks for your attention.. and your effort in resolve questions..

(and, please, excuse my bad english.. as you can see this is not my maternal language..!!!)

good regards,
carlos

Hi Carlos. Your idea of a shell script that recursively calls another shell script could work, but you'd have a lot of processes running, enough that it might well crash your system before you get the million files you seek.

This may be very time-consuming. After you've
done it the first time, try doing a *physical*
backup (i.e., a backup which doesn't need to
perform the system "open()" call for each file.
Then do a restore.
1. If the restore runs faster than the original
procedure of creating the files, use the backup
for future needs.
2. Depending on how your tests process the
files, you might be able to save even more time
by restoring only the directory/inodes.

Just a clarification, Dave.

On Linux, you have to use the -i, or df -i, switch to show inode information.

I can't find the equivalent on Solaris 8, 9 or 10.

yes | head -c 10m | split -b 1

Will give you 10 million 1 byte files.

on Solaris
df -F ufs -o i

An easier way to do this is to use the dd command with a for loop to create many files at the command line. Example below creates a million files each 1 byte each.

for((X=0;X<1000000;X+=1)); do dd if=/dev/zero of=file$X.log bs=1 count=1; done

http://www.fusebit.com/view/post:570