We were being harassed (read: annoyed) by rsync's incremental and non incremental backups for our primary research storage, accounting for ~66 TB.
We were able to reduce run times by 80 to 90% by using GNU parallel and rsync. In a nutshell, a wildcard directory list is created for rsync, which is then piped to GNU parallel.
We were able to reduce run times by 80 to 90% by using GNU parallel and rsync. In a nutshell, a wildcard directory list is created for rsync, which is then piped to GNU parallel.