A commenter of my robocopy article asked me if a multithreaded copy tool like robocopy is really faster then a single threaded copy tool like xcopy as normally the harddisk is the bottleneck and not the CPU when copying files. As I did not found any benchmarks, I decided to do my own benchmark. Here are the results:

In my test, I copied a folder hierarchy with 429 directories and 6313 files with a overall size of 522 MB. I've run every test 2 times to ensure that the speed differences are not caused by filesystem caching. I know that this test was not made under scientific conditions, so if you have your own results, feel free to post them here as comment.

xcopy

Command:

xcopy D:\server\test D:\tmp\test /D /E /Y /Q

Time to copy files

First run: 42 seconds
Second run: 41 seconds

robocopy with 1 thread

Command:

robocopy D:\server\test D:\tmp\test /MT:1 /E /LOG:d:\tmp\robocopy.log

Time to copy the files

First run: 43 seconds
Second run: 41 seconds

robocopy with 10 threads

Command:

robocopy D:\server\test D:\tmp\test /MT:10 /E /LOG:d:\tmp\robocopy.log

Time to copy the files

First run: 38 seconds
Second run: 36 seconds

Conclusion: Using a multithreaded copy tool like robocopy with 10 threads speeds up file copying and makes sense if you have to copy a large number of files. At least on my system.

Robocopy single- and multithreaded benchmark on Windows 7
Tagged on:

17 thoughts on “Robocopy single- and multithreaded benchmark on Windows 7

  • Permalink

    Thanks for the test results. The real question I think is the effect the two copy programs and varied thread counts have on whatever it is I’m doing while the copy is happening? Both background and foreground processes can have profound effects on disk I/O. Also, I am guessing that your copy was single disk. I would be interesting to see the results between two disks, both local, both networked, and mixed..

    Again thanks for the informaton.

    Reply
  • Permalink

    I find the MT option to completely kill copy times. I am mirroring a directory with 216 and it takes at least a minute before the copy even begins. I’m sure I’m getting better throughput during the actual copy process, but the preparation phase is just way too long. I tried a 100,000 file mirror and it just never started at all.

    Reply
  • Permalink

    correlation is not causation. The MT assumption in your tests is slightly flawed. I’d like to see the actual reported threads used by robocopy . I’ve found very little performance gain using the /MT switch and in fact, have found the opposite. Also, how many times was the test run, the results might not pan out on stddev.

    Reply
  • Permalink

    We ran some tests comparing Robocopy with and without the /MT (multithread) option with local disk, 1GbE network and 10GbE network. We tested a variety of numbers and sizes of files. I gave a presentation at our local user group meeting, and the presentation is available at http://www.demartek.com/Demartek_Presenting_RMWTUG_March_2011-03.html. The /MT option provides improved performance and in some cases, significantly improved performance.

    Reply
  • Permalink

    How does the multi threaded option work? What exactly is it doing with the other threads? I hope it isn’t running a number of copy processes simultaneously therefore interleaving the copied files on the target disk.

    Reply
  • Permalink

    Arriving a little late to this thread, but I think your test results might show different results if you measure the time taken to complete a ‘mirror’ or ‘sync’ of two directories. Your tests are just really testing disk throughput as pointed out. Your slight speed increase will be from keeping the disk queue full on the multithreaded tests.

    I think you will find a HUGE improvement when running robocopy in mirror mode, robocopy is checking each file serially to compare size/timestamps etc to work out *IF* it needs to copy the file over. During these checks there is little disk I/O.

    Now running these types of checks in multithreaded mode will see huge speed increases to the overall job time, as its checking 8 (more if you increase the MT count) files at a time, rather than one.

    While this won’t make the disk throughput any faster, it will allow one ‘thread’ to do a disk copy while the other 7 threads continue on checking for file changes. Resulting in a reduced job time for mirrored jobs not requiring a a huge data copy over the wire.

    Reply
  • Permalink

    Also, try this to copy files over the network. I’ve found that using windows copy to copy a number of smaller files is extremely slow, and it looks like most of the wait is for some sort of handshake thing between the local and remote computers, so theoretically, if you have 10 of these operations going at the same time, the operation could be much faster.

    Reply
  • Permalink

    Nice comparison, through the years I’m using Long Path Tool it works great maybe we could also try this as an alternative

    Reply
  • Permalink

    I work as a sysadmin at an AIDS research center and I copy huge amounts of data every data. I’ve found that multi-threaded robocopy improved speed a lot, and I mean it, A LOT. On a Xeon Sandy Bridge 8-core machine, the MT option in Robocopy improved speed by a little over 60% when copying over to a different disk (not over the network). This on a dataset with some hundred thousands of small files.

    Reply
  • Permalink

    Can I put more threads than physical cores I have?

    Reply
  • Permalink

    You can try another free portable file copier software Exshail CopyCare from

    below site. Main feature is Preview list of files before copying with seven

    options below.

    1. “Source > Target – Copy Files New and changed from Source”
    2. “Source > Target – Copy Files New From Source”
    3. “Source > Target – Copy Files Changed from Source”
    4. “Target > Source – Copy Files Changed from Target”
    5. “Target Source Copy Files having Size Difference”
    6. “Delete Files Orphan from Target”
    7. “Source = Target – Copy Exact to Target – Overwrite All (Delete Orphans

    from Target)”

    https://sites.google.com/site/exshail/exshailcopycare

    Reply
  • Permalink

    I backed up 340Gb (approx. 62500 files of varying size) using Robocopy from one internal SATA drive to another internal SATA.

    Using /MT:32 it took 11 hrs 37 mins (yes almost 12 hours). After testing several different values for the MT switch agains a smaller file set it seemed completely omitting the switch resulted in the quickest throughput.

    The same 340Gb backup on the same hardware was then carried out without /MT… 1 hr 44 mins!

    My experience would suggest leaving out /MT if copying across internal disks on the same system.

    Reply
    • Permalink

      Thank You! we found the same results.. the MT switch definitely slowed down the copy process between internal disks on the same system.

      Reply
      • Permalink

        It depends on the types of files you’re copying, and the drive technology you have. If you’re using conventional hard drives and copying large files, then yeah, performance will suffer due to the head having to move for each file it’s copying. But, if it’s a bunch of smaller files, /MT would help a lot. It also depends on how fragmented your drives are.

        Now, if you’re using SSD’s, you won’t have head seeks getting in your way, and you’ll likely see a small increase in performance using /MT for any file size. It all depends on the size of the buffer being used by RoboCopy and how fast it’s reading/writing per thread.

        Reply
    • Permalink

      Yes Sata to Sata, HDD or SSD? /MT takes 8 cores by default. how many did you have available?
      So you have 8 cores, cool that makes 8 cores collecting the files together from the same HDD and copy them to another HDD on the same controller?
      Where is the bottleneck? don’t blame robocopy.

      Please try Robocopy from SSD to SSD on a different controller, make sure you have enough cores available, stay 2 under the max available.
      or cool: copy data with robocopy over a 10Gbe (copper or fiber) to your network storage.
      Make Robocopy great again. This really works.

      SvE
      IT

      Reply
  • Permalink

    Robocopy replaces XCopy in the newer versions of windows

    Uses Mirroring, XCopy does not
    Has a /RH option to allow a set time for the copy to run
    Has a /MON:n option to check differences in files
    Copies over more file attributes than XCopy

    The most important difference is that robocopy will (usually) retry when an error occurs, while xcopy will not. In most cases, that makes robocopy far more suitable for use in a script.
    If you have XP or Windows Server you can easily get this in the Resource Kits. If you have Vista, it’s already in your path. That’s always nice. It’s Robust, indeed (hence, Robocopy) but it’s legendarily unforgiving. If anything is wrong with the command line options you’ll just get the help. It’s so hard to use there’s even a GUI Frontend you can get. However, when I want to get a directory from here to over there, I just do this (no wildcards allowed! Doh!) and it just gets there, auto skipping files that are already at the destination. It’s also wonderful over an unreliable network: More on robocopy…..

    http://net-informations.com/q/mis/robocopy.html

    robocopy “H:\Source” “z:\Dest” /S /Z

    Where /s means subdirectories, and /z means in restartable mode.

    Lee

    Reply
  • Permalink

    I suggest you to download new Long Path Tool software that simply allows you to work easily on Long Path files.

    Thank you.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Ad #native_company# — #native_desc# #native_cta#