Robocopy single- and multithreaded benchmark on Windows 7
A commenter of my robocopy article asked me if a multithreaded copy tool like robocopy is really faster then a single threaded copy tool like xcopy as normally the harddisk is the bottleneck and not the CPU when copying files. As I did not found any benchmarks, I decided to do my own benchmark. Here are the results:
In my test, I copied a folder hierarchy with 429 directories and 6313 files with a overall size of 522 MB. I’ve run every test 2 times to ensure that the speed differences are not caused by filesystem caching. I know that this test was not made under scientific conditions, so if you have your own results, feel free to post them here as comment.
xcopy
Command:
xcopy D:\server\test D:\tmp\test /D /E /Y /Q
Time to copy files
First run: 42 seconds
Second run: 41 seconds
robocopy with 1 thread
Command:
robocopy D:\server\test D:\tmp\test /MT:1 /E /LOG:d:\tmp\robocopy.log
Time to copy the files
First run: 43 seconds
Second run: 41 seconds
robocopy with 10 threads
Command:
robocopy D:\server\test D:\tmp\test /MT:10 /E /LOG:d:\tmp\robocopy.log
Time to copy the files
First run: 38 seconds
Second run: 36 seconds
Conclusion: Using a multithreaded copy tool like robocopy with 10 threads speeds up file copying and makes sense if you have to copy a large number of files. At least on my system.
Thanks for the test results. The real question I think is the effect the two copy programs and varied thread counts have on whatever it is I’m doing while the copy is happening? Both background and foreground processes can have profound effects on disk I/O. Also, I am guessing that your copy was single disk. I would be interesting to see the results between two disks, both local, both networked, and mixed..
Again thanks for the informaton.
I find the MT option to completely kill copy times. I am mirroring a directory with 216 and it takes at least a minute before the copy even begins. I’m sure I’m getting better throughput during the actual copy process, but the preparation phase is just way too long. I tried a 100,000 file mirror and it just never started at all.
correlation is not causation. The MT assumption in your tests is slightly flawed. I’d like to see the actual reported threads used by robocopy . I’ve found very little performance gain using the /MT switch and in fact, have found the opposite. Also, how many times was the test run, the results might not pan out on stddev.
We ran some tests comparing Robocopy with and without the /MT (multithread) option with local disk, 1GbE network and 10GbE network. We tested a variety of numbers and sizes of files. I gave a presentation at our local user group meeting, and the presentation is available at http://www.demartek.com/Demartek_Presenting_RMWTUG_March_2011-03.html. The /MT option provides improved performance and in some cases, significantly improved performance.
How does the multi threaded option work? What exactly is it doing with the other threads? I hope it isn’t running a number of copy processes simultaneously therefore interleaving the copied files on the target disk.
Arriving a little late to this thread, but I think your test results might show different results if you measure the time taken to complete a ‘mirror’ or ‘sync’ of two directories. Your tests are just really testing disk throughput as pointed out. Your slight speed increase will be from keeping the disk queue full on the multithreaded tests.
I think you will find a HUGE improvement when running robocopy in mirror mode, robocopy is checking each file serially to compare size/timestamps etc to work out *IF* it needs to copy the file over. During these checks there is little disk I/O.
Now running these types of checks in multithreaded mode will see huge speed increases to the overall job time, as its checking 8 (more if you increase the MT count) files at a time, rather than one.
While this won’t make the disk throughput any faster, it will allow one ‘thread’ to do a disk copy while the other 7 threads continue on checking for file changes. Resulting in a reduced job time for mirrored jobs not requiring a a huge data copy over the wire.
Also, try this to copy files over the network. I’ve found that using windows copy to copy a number of smaller files is extremely slow, and it looks like most of the wait is for some sort of handshake thing between the local and remote computers, so theoretically, if you have 10 of these operations going at the same time, the operation could be much faster.