top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Fastest way of removing very large number of files?

+2 votes
392 views

I currently have a problem on my backup server with very large number of small files in a large number of directories. I would like to delete them as fast as possible. Currently I use:

rsync -a --delete /empty_directory/ dir_to_clean/

I've read that rsync will be faster than rm or find. Can someone recommend something? I use an ext4 filesystem.

posted Aug 23, 2013 by Amit Parthsarthi

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button

3 Answers

+2 votes

A much faster way of deleting files – the “find” command has a “-delete” flag built right in!!

[root@localhost Desktop]# find . -type f -delete

Using this method, I was deleting files at a rate of about 2000 files/second.

You can also show the filenames as you’re deleting them:

[root@localhost Desktop]# find . -type d -print -delete

Or even show how many files will be deleted, then time how long it takes to delete them:

[root@localhost Desktop]# ls -1 | wc -l && time find . -type f -delete
real    0m3.660s
user    0m0.036s
sys     0m0.552s
answer Aug 30, 2013 by Satyabrata Mahapatra
+1 vote

I suspect that a C program that calls remove() would probably be faster than just about anything else.

answer Aug 23, 2013 by Jagan Mishra
I agree with what you are saying, and I think you may even want to use unlink() if they are all files, no dirs. remove() actually will call unlink() so you may be able to save few function calls and logical statements.
+1 vote

If it's on a different file system than mkfs is the fastest. ;)

answer Aug 23, 2013 by Kumar Mitrasen
Similar Questions
+1 vote

Is there is a opposite of __init__.py like __del__.py

I want, that when the application ends, certain functions are executed. I know I could make a constructor and a destructor, but I simply want to know if there is a opposite....

+2 votes

I use Apache server on a Arch Linux machine. I have a big problem with the delete requests. Apache refuse all of them with the 403 response status code.

I do not understand why. Please help me?

+1 vote

When I run git clean -xfd, git deletes my tags file. I have the tag file listed in gitignore, but how do I tell git not to remove the tags file.

0 votes

MONGODB 3.2.8, WIREDTIGER STORAGE ENGINE
SINGLE COLLECTION HAS MORE THAN 500G, WHETHER IT NEEDS TO BE SEPARATED?

WHAT PARAMETER SHOULD BE ADJUSTED ?

...