How to quickly delete millions of files

  • A+
Category:Languages

We use BOOST1.63 boost::filesystem::remove_all(dir_to_remove) to remove a folder that has millions of files (each file has size of 1MB). The folder "dir_to_remove" has sub-folders, and each sub-folder has no more than 1000 files. It takes more than 10-min to delete all files. We use CentOS6.5.

After checking the operations.cpp, we realized BOOST actually uses Linux rmdir and unlink commands:

#   define BOOST_REMOVE_DIRECTORY(P)(::rmdir(P)== 0) #   define BOOST_DELETE_FILE(P)(::unlink(P)== 0) 

This article listed several ways to delete files more efficiently on Linux. And it recommended to use rsync.

How can we delete millions of files quickly with C++?


If you want to get free a required location, the fastest way to do it is moving (or renaming) a directory to another location on the same partition. Then your program can continue work with a required location and remove a previously moved directory recursively in another thread (in background). This thread can even work with less priority, so removing a specific directory would look like an instant filesystem operation.

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: