Copy on file with multithread

[ChristoheB]

Hi,

I tested more ssh's options. And here are the results :
Command : scp OPTIONS /test.img root@192.168.1.1:/dev/null
pas d'option => 145 Mo/s
-c none => 145 Mo/s
-c 3des-cbc => 23 Mo/s
-c blowfish-cbc => 87 Mo/s
-c arcfour => Lost connexion
-c aes256-ctr => 175 Mo/s
-c aes128-ctr => 210 Mo/s
Yes, if i change the Cipher then it's better.

But the maximum is 210 Mo/s and i'm looking for 1500Mo/s. I understand that if i use nfs, nbt or nc, then i can obtain 1500. But i'm looking for about ssh (native).

[TrevorH]

So, as said, the problem is not with copying nor your network, it's to do with encrypting the data. If you want it faster then you don't have many choices:

1) use something other than ssh. You say it has to be "Native" but NFS is that too.
2) use a worse cipher though on your setup it appears that nothing is very good so I am guessing you have an old slow processor, possibly without built-in AES support

If you currently get 200MB/s and want to get to 1500 then you will need to make drastic changes to the way you do this. An increase of 100% throughput might be possible but one of 800%? Not likely, not without dropping encryption and transferring in plain text. Replacing your processor with a faster one that has AES native instructions might help but I doubt if it'll get you 8 times your current throughput.

[ChristoheB]

Thank you

You right, it's an old cpu. But i can't change it because i works with about 400 Servers that is why i'm looking for an another solution.
You right, i think that use ssh without encryption is my solution, but I use CentOS 7.7 and i think that the Chiper none is not compilate by default.I am currently working on it.

[TrevorH]

No, you will need to use something OTHER than ssh.

[chemal]

No, you will need to use something OTHER than ssh.

Good old rcp gives me 910 MB/s over a 10 GBit/s IPoIB link with 54% load on a single core of a 2008 i7-920.

[drk]

When I transfer files I use pigz to compress the data and then pipe that through ssh

Code: Select all

$ tar cf - filename(s) | pigz | ssh -oCompression=no user@remote "cd /destination/directory && tar xvzf -"

Usually more than one file, so tar, but tar also keeps permissions etc on the receiving machine. That gets more threads on the sending machine but still is single threaded on receiving machine.

[KernelOops]

In you want to run multiple scp commands (or other commands for that matter) then you may use the awesome good old parallel command:

Code: Select all

NAME
       parallel - build and execute shell command lines from standard input in parallel

SYNOPSIS
       parallel [options] [command [arguments]] < list_of_arguments

       parallel [options] [command [arguments]] ( ::: arguments | :::: argfile(s) ) ...

       parallel --semaphore [options] command

       #!/usr/bin/parallel --shebang [options] [command [arguments]]

[jlehtone]

In you want to run multiple scp commands

There are many ways to run multiple commands. However, that helps (here) only if there are multiple files to transfer.