[UCI-Linux] parsync

harry mangalam harry.mangalam at uci.edu
Wed Apr 2 11:40:51 PDT 2014


> I'm curious, what would be the usecase for using parsync instead of, say,
> GNU parallel?  (Aside from the obvious of being purpose-written.)  I use
> parallel for running multiple imapsync jobs when we migrate users from
> IMAP->Exchange, and I recall that it has lots of applicable functions. 
> IIRC it also has pieces that seem directly applicable to coordinating
> multi-host functions, although I haven't played with those.

gnu parallel will fork off multiple instances of rsync fine, but that won't 
help when you're trying to loadbalance.

ie parallel will fork X rsyncs which will all finish at different times unless 
you manually load-balance the input files (which parsync takes care of). 
Actually parsync's child rsyncs will finish at sligtly different times as well 
since the actual time to complete depends not only on size, but also the 
number of files (transferring 1GB in 1 file is significantly faster than 
transferring 1GB in a million files - we frequently run into that situation).

Also, a recursion into a very large filesystem can take hours.  parsync will 
cache the results of that recursion and allow you to reuse that data, so if 
something happens that interrupts the rsync, it doesn't have to recurse again.

The multi-host features of parallel, while very nice, aren't really applicable 
to this kind of rsync.

---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillists.uci.edu/pipermail/uci-linux/attachments/20140402/61f2360b/attachment.html>


More information about the UCI-Linux mailing list