LinHES Forums
http://forum.linhes.org/

Speed-up rsync and scp xfers on a LAN
http://forum.linhes.org/viewtopic.php?f=24&t=22494
Page 1 of 1

Author:  graysky [ Wed Feb 29, 2012 3:58 pm ]
Post subject:  Speed-up rsync and scp xfers on a LAN

Just wanted to post the results of my analysis comparing different ciphers on scp throughput on "modern" hardware.

Summary
After measuring how long various v2 ciphers took to transfer a 500 MB on a Gigalan network, either arcfour, arcfour128, or arcfour256 gave the fastest result on modern hardware.

Experiment
Using scp, a 500 MB file from computer A to computer B was transferred (5 times totally) over my LAN and the results were timed. A different 500 MB file for each test was generated from /dev/urandom for each of the trials to avoid any disk caching trickery. It is noteworthy to mention that most of the ciphers were CPU bound; in other words, watching htop throughout the experiment, most of the ciphers cause one core to hit 100 % usage. These have been denoted in the data with a pink color. A blue color shows those ciphers that did not max out any single core.

Hardware details
Both computer A and computer B are Intel P45-based boards. Computer A is an Intel X3360 @ 3.40 GHz and computer B is an Intel E5200 @ 3.33 GHz. Both machines had their CPU governors set to 'performance' for this test. Both machines have Gigalan NICs (onboard) using 4k jumbo frames. The LAN itself iuses a gigalan switch (Cisco SR2016T-NA) and is comprised entirely of CAT5E cables (wired the house myself). Hardrive access time are irrelevant in this test since transfers are to/from tmpfs (/scratch is mounted to tmpfs).

Results
Image
Larger throughputs are desired.

Image
Smaller times are desired.

The variation between the 5 runs is pretty tight on the whole; the standard errors are plotted on the bar graphs. There is no statistically significance difference between the arcfour, arcfour128, and arcfour256 ciphers in this test as seen by an ANOVA analysis on the five runs/cipher:
Image

Methods and Data
The raw data are here: http://pastebin.com/uqHaM72q

The script is here:
Code:
for cipher in arcfour arcfour128 arcfour256 aes128-ctr aes192-ctr aes256-ctr aes128-cbc 3des-cbc blowfish-cbc cast128-cbc aes192-cbc aes256-cbc; do
  echo "$cipher"

  for try in 1 2 3 4 5; do
    dcfldd statusinterval=50 if=/dev/urandom of=/scratch/random bs=1M count=500
   
    start=$(date +%s.%N)
    scp -c "$cipher" /scratch/random facade@mars:/scratch
    end=$(date +%s.%N)
    diff=$(echo "scale=6; $end - $start" | bc)

    [[ ! -f /scratch/xfer.log ]] && echo "cipher,try,time(sec)" > /scratch/xfer.log
    echo "$cipher,$try,$diff" >> /scratch/xfer.log
  done
done

Author:  Liv2Cod [ Wed Feb 29, 2012 8:13 pm ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

Oooo... Good lead-up. I really want to see that "image" now! Please, please post it!

Author:  mattbatt [ Wed Feb 29, 2012 10:24 pm ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

For us slightly more ignorant people.... what do you use this for?

Author:  graysky [ Thu Mar 01, 2012 3:18 am ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

Liv2Cod wrote:
Oooo... Good lead-up. I really want to see that "image" now! Please, please post it!

:confused: -- what image?

mattbatt wrote:
For us slightly more ignorant people.... what do you use this for?

If you transfer files around on your LAN via scp or rsync -e ssh selecting arcfour128 as the cipher method will give superior xfer speeds. If you use NFS or samba for sharing files, this is not relevant to you.

Author:  christ [ Thu Mar 01, 2012 7:12 am ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

graysky wrote:
If you transfer files around on your LAN via scp or rsync -e ssh selecting arcfour128 as the cipher method will give superior xfer speeds.


graysky,
have you determined whether this is because you are CPU bound (ie. encrypt/decrypt is slowing you down) vs the resulting encrypted payload being bigger?

Given the names I would expect a 128 bit cipher to be lighter but am curious where you were seeing the bottlenecks.

Author:  marc.aronson [ Thu Mar 01, 2012 11:25 am ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

He's running with fairly beefy machines -- I doubt he is CPU bound, but it's a good question...

Author:  graysky [ Thu Mar 01, 2012 3:05 pm ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

I can test the CPU-bound question in a few hours... I think you're asking if a processor is pegged using the less efficient ciphers while the xfer is active?

Author:  bobmyth [ Thu Mar 01, 2012 9:20 pm ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

I was getting 1.25MB/s transfers with an old SPARC system at work, the normal bandwidth on the LAN is around 60MB/s. I just changed today to arcfour128 and the transfer rate is up to 2.0MB/s. In this case, I'm guessing it was CPU bound.

Author:  christ [ Thu Mar 01, 2012 9:39 pm ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

graysky wrote:
I can test the CPU-bound question in a few hours... I think you're asking if a processor is pegged using the less efficient ciphers while the xfer is active?

Not exactly. It may be that the encode/decode cycle takes longer than the serial line speed of your ethernet port. So it may not peg your CPU.

I'm curious because my vague recollection of the RC4 algorithm is that it is a stream cipher which should not inflate the size of the payload except maybe the smallest of packets. RC4 is also pretty light weight processor-wise so I wouldn't think that would be it either. Hence my confusion.

Perhaps it is some sort of transfer delay. no idea really. just shooting in the dark.

Author:  graysky [ Fri Mar 02, 2012 5:32 am ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

christ wrote:
graysky wrote:
have you determined whether this is because you are CPU bound (ie. encrypt/decrypt is slowing you down) vs the resulting encrypted payload being bigger?


Excellent question. Only 4 of the chipers were NOT CPU bound: arcfour, arcfour128, arcfour256, and aes128-cbc. I also re-ran the analysis setting the CPU scaling governors on both machines to 'performance' and will edit the first post shortly with the results.

Author:  christ [ Fri Mar 02, 2012 7:02 am ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

graysky wrote:
christ wrote:
graysky wrote:
have you determined whether this is because you are CPU bound (ie. encrypt/decrypt is slowing you down) vs the resulting encrypted payload being bigger?


Excellent question. Only 4 of the chipers were NOT CPU bound: arcfour, arcfour128, arcfour256, and aes128-cbc. I also re-ran the analysis setting the CPU scaling governors on both machines to 'performance' and will edit the first post shortly with the results.

And I should have better noted earlier that you said there was no significant difference between arcfour128 and arcfour256. That makes more sense to me. As I mentioned RC4 is pretty lightweight and incidentally most view it as not particularly secure if that matters to you.

Note too that most of the other ciphers are block ciphers and will have some inflation of the payload due to padding because the blocks have to be aligned with the cipher size.

Author:  Liv2Cod [ Sat Mar 03, 2012 12:56 pm ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

graysky wrote:
Results
Image
Larger throughputs are desired.

Image
Smaller times are desired.


I just thought there may be some pretty pictures of this instead of the word "image"

Author:  graysky [ Sun Mar 04, 2012 2:35 am ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

:confused: I see images... you don't?

EDIT: Very odd. I uploaded png files. I use chromium which displays them just fine. When I browse to this thread using firefox, they are not there. I edited the first post replacing the png files with jpg files. Now firefox renders them. Thanks for letting me know.

Author:  graysky [ Sun May 13, 2012 1:15 pm ]
Post subject:  Re: Speed-up rsync and scp xfers on a LAN

I should add that users need not specify the cipher in the scp or rsync command, simple do it on a per-hosts basis in ~/.ssh/config on the machine that will SEND the files:

Code:
Host  mars
Ciphers arcfour128

Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/