From harry.mangalam at uci.edu Thu Nov 5 09:13:42 2009 From: harry.mangalam at uci.edu (Harry Mangalam) Date: Thu, 5 Nov 2009 09:13:42 -0800 Subject: [UCCSC] What google knows about you and how you can change it .. maybe... Message-ID: <200911050913.42465.harry.mangalam@uci.edu> This may be of general interest for users of google services. http://googleblog.blogspot.com/2009/11/transparency-choice-and-control-now.html -- Harry Mangalam - Research Computing, NACS, Rm 225 MSTB, UC Irvine [ZOT 2225] / 92697 949 824-0084(o), 949 285-4487(c) MSTB=Bldg 415 (G-5 on --- It is better to be roughly right than precisely wrong. Keynes From harry.mangalam at uci.edu Mon Nov 9 09:47:23 2009 From: harry.mangalam at uci.edu (Harry Mangalam) Date: Mon, 9 Nov 2009 09:47:23 -0800 Subject: [UCCSC] NAS followup Message-ID: <200911090947.24110.harry.mangalam@uci.edu> Two more data points may be of interest. 1 - Drobo Pro ------------- I had an opportunity to play with a Drobo Pro, an interesting 8-slot NAS device that uses a proprietary RAIDing system that allows transparent hot disk swapping, different disk sizes, iSCSI, firewire-800 and USB-2, and many other options: It was bought to provide temp space for a Next-Gen Genomic Sequencing machine that will generate about 4TB raw data per run and requires parallel processing on a Linux platform to reduce the raw data to usable sequence. Unfortunately, the Drobo Pro can only be set up with Windows/Mac, so we tried to use a virtualized Windows (via VMware) to set it up. That failed, as the networking would not allow us to share the Drobo with any external hosts. So Windows was used as the host OS, with Linux as the guest OS. That allowed the Drobo to be shared, but then the guest Linux OS was only able to use 2 of the 16 cores, less than optimal. Ahhh, the pleasures of layered technology. Also unfortunately, while it does support Linux filesystems (ext2/3 only) it only supports partitions up to 2 TB in size. This may not be a problem for many, but it is for this use case, since a single partition can't hold the data for even 1 sequencing run. It also has a pretty slow performance via USB-2 (didn't try firewire), so while it can store the data, it's too slow to act as an actual data feed when connected to the processing machine. After discovering that it could only provide 2 TB partitions to Linux, we did not explore the iSCSI connection, which should have provided much higher bandwidth. For a Windows-only application, it may be a good approach, but there are too many mismatches for a good Linux solution. 2 - Backblaze - 67TB for <$8K One of the Genome core people sent me this link which is essentially a homemade Sun Thumper (aka Sunfire 4540 ) for ~ 1/4 the price: Backblaze has made it their default storage block, so they're essentially betting their farm on it, and thus by definition, it is "enterprise storage". It's not super-fast, but it is super cheap and apparently, quite reliable using JFS on RAID6 via software (mdadm). Unless it lacked the chipset driver support, they could also have used Open Solaris and ZFS on it as well) This is essentially the way I've provided a number of (much smaller) NAS devices after becoming disenchanted with various commercial NASs. The much higher performance approach is to use hardware controllers (3ware/Areca, etc) which eases setup, provides battery-backed cache, and increases performance quite a bit, but is much more expensive (~$2200 for 48 ports of HW controllers vs ~$175 for the dirt-cheap Syba softRAID controllers). It you're considering a disk-only backup solution, something like this may be of interest. -- Harry Mangalam - Research Computing, NACS, Rm 225 MSTB, UC Irvine [ZOT 2225] / 92697 949 824-0084(o), 949 285-4487(c) MSTB=Bldg 415 (G-5 on --- It is better to be roughly right than precisely wrong. Keynes From cabeen at chem.ucsb.edu Mon Nov 9 14:21:08 2009 From: cabeen at chem.ucsb.edu (Ted Cabeen) Date: Mon, 09 Nov 2009 14:21:08 -0800 Subject: [UCCSC] NAS followup In-Reply-To: <200911090947.24110.harry.mangalam@uci.edu> References: <200911090947.24110.harry.mangalam@uci.edu> Message-ID: <4AF895D4.10107@chem.ucsb.edu> The backblaze is cool, but it definitely costs more than $8k if you factor in the time and effort they went to to build the custom cases they're using. Because they built a bunch of these machines, it makes sense for them, but since you can't buy their custom case anywhere right now, it's not really practical. If they sold the case and the harder-to-find parts for their system, it might be worth it, but until then, it doesn't seem like a feasible solution for smaller systems. --Ted On 11/9/2009 9:47 AM, Harry Mangalam wrote: > 2 - Backblaze - 67TB for<$8K > > One of the Genome core people sent me this link which is essentially a > homemade Sun Thumper (aka Sunfire 4540 > ) for ~ 1/4 the price: > > > Backblaze has made it their default storage block, so they're > essentially betting their farm on it, and thus by definition, it > is "enterprise storage". It's not super-fast, but it is super cheap > and apparently, quite reliable using JFS on RAID6 via software > (mdadm). Unless it lacked the chipset driver support, they could > also have used Open Solaris and ZFS on it as well) > > This is essentially the way I've provided a number of (much smaller) > NAS devices after becoming disenchanted with various commercial NASs. > > The much higher performance approach is to use hardware controllers > (3ware/Areca, etc) which eases setup, provides battery-backed cache, > and increases performance quite a bit, but is much more expensive > (~$2200 for 48 ports of HW controllers vs ~$175 for the dirt-cheap > Syba softRAID controllers). > > It you're considering a disk-only backup solution, something like this > may be of interest. > From harry.mangalam at uci.edu Mon Nov 9 16:03:51 2009 From: harry.mangalam at uci.edu (Harry Mangalam) Date: Mon, 9 Nov 2009 16:03:51 -0800 Subject: [UCCSC] NAS followup In-Reply-To: <4AF895D4.10107@chem.ucsb.edu> References: <200911090947.24110.harry.mangalam@uci.edu> <4AF895D4.10107@chem.ucsb.edu> Message-ID: <200911091603.51783.harry.mangalam@uci.edu> Good points - it certainly cost them more than that to start with, but it paid off for them. There is a fair amount of interest in the process. If you ramble over to: there is more discussion about as well as pointers to a company who will take the contributed Solidworks CAD file and turn it into sheet metal for you (tho probably not at a one-off rate that will give you $8K/100TB). There is the expected carping from 'people who know better' and some of them have good points. There are other similar solutions available, tho none at that size/pricepoint, which was the reason Backblaze designed and built it. They obviously feel confident enough to release the spec for public comment and implementation. It will be interesting to see how the design evolves over time. As I noted, I'd want to replace the spaghetti of SW RAID controllers and port-multipliers with decent multilane SATA controller with BBU, but others might not want to. There is a slowly enlarging Open Source Hardware movement, so someone may decide to build and market this in the not-too-distant future a la Arduino and its (legal) copycats: And ICs are a lot harder than sheet metal. I suspect that we'll see chassis' like these within a few months at very competitive prices. It would still be pretty competitive if it was double the price. Note that this isn't a suggestion to use these in place of NetApps and EMC drives, but if you need vast, fairly slow, but fairly reliable storage, it might be useful (like staging storage for the Genome Sequencer, disk-based backup servers, scratch space for very large data (LHC data, medical image data sets, etc) hjm On Monday 09 November 2009 14:21:08 Ted Cabeen wrote: > The backblaze is cool, but it definitely costs more than $8k if you > factor in the time and effort they went to to build the custom > cases they're using. Because they built a bunch of these machines, > it makes sense for them, but since you can't buy their custom case > anywhere right now, it's not really practical. > > If they sold the case and the harder-to-find parts for their > system, it might be worth it, but until then, it doesn't seem like > a feasible solution for smaller systems. > > --Ted > > On 11/9/2009 9:47 AM, Harry Mangalam wrote: > > 2 - Backblaze - 67TB for<$8K > > > > One of the Genome core people sent me this link which is > > essentially a homemade Sun Thumper (aka Sunfire 4540 > > ) for ~ 1/4 the price: > > >o-build-cheap-cloud-storage/> > > > > Backblaze has made it their default storage block, so they're > > essentially betting their farm on it, and thus by definition, it > > is "enterprise storage". It's not super-fast, but it is super > > cheap and apparently, quite reliable using JFS on RAID6 via > > software (mdadm). Unless it lacked the chipset driver support, > > they could also have used Open Solaris and ZFS on it as well) > > > > This is essentially the way I've provided a number of (much > > smaller) NAS devices after becoming disenchanted with various > > commercial NASs. > > > > The much higher performance approach is to use hardware > > controllers (3ware/Areca, etc) which eases setup, provides > > battery-backed cache, and increases performance quite a bit, but > > is much more expensive (~$2200 for 48 ports of HW controllers vs > > ~$175 for the dirt-cheap Syba softRAID controllers). > > > > It you're considering a disk-only backup solution, something like > > this may be of interest. -- Harry Mangalam - Research Computing, NACS, Rm 225 MSTB, UC Irvine [ZOT 2225] / 92697 949 824-0084(o), 949 285-4487(c) MSTB=Bldg 415 (G-5 on --- It is better to be roughly right than precisely wrong. Keynes From harry.mangalam at uci.edu Fri Nov 20 14:24:39 2009 From: harry.mangalam at uci.edu (Harry Mangalam) Date: Fri, 20 Nov 2009 14:24:39 -0800 Subject: [UCCSC] File Synchronizer for MacOS X Message-ID: <200911201424.39153.harry.mangalam@uci.edu> Is there a good, free, OSS, or commercial file synchronizer for the Mac? Or a verified web app that does the same? A (surprisingly) data-security-conscious client wants to use a USB disk to sync his desktop to his laptop at home and has tried a number of apps that do not work well. He was very keen on the commercial service Sugar-Sync until it started to frag his data and required regular manual cleanups via their tech service. He has since soured on Sugar. time machine seems to be a versioning app rather than a synchro app. I haven't used it, but he has and claims that it doesn't do 2way syncs. He's tried to use Bombich's Carbon Copy Cloner, but it does not (and will not) allow 'remote to local' sync'ing bc they don't trust users to avoid overwriting critical boot disk files. I set him up with an crontabbed rsync backup for regular sync's to a remote server, which he's quite happy with, but he's not a CLI geek and has already come close to wiping out his archives by playing with the options, so I'm reluctant to set up such a system for him to do this with naked rsync. I tried Unison for him (this is what it was designed to do) and while the GUI is simple, it works very well and does allow both local 2-way syncing (as to a USB disk) and remotely (via a number of protocols). It also has a native Aqua interface, but there's no Snow Leper finkable version. Are there any other obvious-to-a-Mac-user choices that I've missed? hjm -- Harry Mangalam - Research Computing, NACS, Rm 225 MSTB, UC Irvine [ZOT 2225] / 92697 949 824-0084(o), 949 285-4487(c) MSTB=Bldg 415 (G-5 on --- It is better to be roughly right than precisely wrong. Keynes From grahamp at berkeley.edu Fri Nov 20 14:46:27 2009 From: grahamp at berkeley.edu (Graham Patterson) Date: Fri, 20 Nov 2009 14:46:27 -0800 Subject: [UCCSC] File Synchronizer for MacOS X In-Reply-To: <200911201424.39153.harry.mangalam@uci.edu> References: <200911201424.39153.harry.mangalam@uci.edu> Message-ID: <4B071C43.3060701@berkeley.edu> I have used FileSync to get Mac users to batch copy files to a (Windows) server. It has limitations, such as no exclusions within a folder tree. It also has a mirror function, though I would read up on that part carefully first. http://www.designersdomain.com/filesync/ For Windows people there is always SyncToy. Graham -- Graham Patterson, Systems Administrator Lawrence Hall of Science, UC Berkeley 510-643-2222 From aakash.shah at research.uci.edu Fri Nov 20 15:20:12 2009 From: aakash.shah at research.uci.edu (Aakash Shah) Date: Fri, 20 Nov 2009 15:20:12 -0800 Subject: [UCCSC] File Synchronizer for MacOS X In-Reply-To: <4B071C43.3060701@berkeley.edu> References: <200911201424.39153.harry.mangalam@uci.edu> <4B071C43.3060701@berkeley.edu> Message-ID: <62A181A4DF33074AB71D8A9CB07AFF3F4F8318@email.rgs.uci.edu> SuperDuper and Synchronize X are two other tools to consider: http://www.shirt-pocket.com/SuperDuper/SuperDuperDescription.html http://www.qdea.com/synchronize_x_plus_intro.html Aakash Shah Systems Administrator UCI - Office of Information Technology -----Original Message----- From: uccsc-bounces at uci.edu [mailto:uccsc-bounces at uci.edu] On Behalf Of Graham Patterson Sent: Friday, November 20, 2009 2:46 PM To: Harry Mangalam Cc: uccsc at uci.edu Subject: Re: [UCCSC] File Synchronizer for MacOS X I have used FileSync to get Mac users to batch copy files to a (Windows) server. It has limitations, such as no exclusions within a folder tree. It also has a mirror function, though I would read up on that part carefully first. http://www.designersdomain.com/filesync/ For Windows people there is always SyncToy. Graham -- Graham Patterson, Systems Administrator Lawrence Hall of Science, UC Berkeley 510-643-2222 _______________________________________________ List-Info: https://maillists.uci.edu/mailman/listinfo/uccsc From icrew at berkeley.edu Fri Nov 20 15:23:22 2009 From: icrew at berkeley.edu (Ian Crew) Date: Fri, 20 Nov 2009 15:23:22 -0800 Subject: [UCCSC] File Synchronizer for MacOS X In-Reply-To: <200911201424.39153.harry.mangalam@uci.edu> References: <200911201424.39153.harry.mangalam@uci.edu> Message-ID: <691A5633-AF51-4B3E-927D-1F9BBA644EF1@berkeley.edu> I don't have experience with it, but check out http://www.bartbusschots.ie/blog/?p=338 for a GUI for the Mac OS X rsync.... Ian On Nov 20, 2009, at 2:24 PM, Harry Mangalam wrote: > I set him up with an crontabbed rsync backup for regular sync's to a > remote server, which he's quite happy with, but he's not a CLI geek > and has already come close to wiping out his archives by playing with > the options, so I'm reluctant to set up such a system for him to do > this with naked rsync. Ian Crew Media Vault Program Information Services and Technology-Data Services University of California, Berkeley 2195 Hearst Ave, Second Floor http://mvp.berkeley.edu 2009 Chair, Chancellor's Staff Advisory Committee http://csac.chance.berkeley.edu From medmonds at ucsc.edu Fri Nov 20 16:09:00 2009 From: medmonds at ucsc.edu (Michael P Edmonds) Date: Fri, 20 Nov 2009 16:09:00 -0800 Subject: [UCCSC] File Synchronizer for MacOS X In-Reply-To: <200911201424.39153.harry.mangalam@uci.edu> References: <200911201424.39153.harry.mangalam@uci.edu> Message-ID: <38760AAD-1AA7-4443-93C1-62F4810ADB57@ucsc.edu> Harry -- We use Super Flexible File Synchronizer for the Social Sciences Division at UCSC and are quite pleased with it. As I recall, we were able to get a worldwide site license for the professional edition (Macs and PCs), allowing any UCSC faculty or staff to use it any where in the world for $1,200. One thing to keep in mind is that for Mac users, they must be running Intel-based systems (no PowerPC support). http://www.superflexible.com/ Thanks! -- Mike Michael P Edmonds ITS Divisional Liaison - Division of Social Sciences ITS - EAST 459-3165 medmonds at ucsc.edu On Nov 20, 2009, at 14:24, Harry Mangalam wrote: > Is there a good, free, OSS, or commercial file synchronizer for the > Mac? Or a verified web app that does the same? > > A (surprisingly) data-security-conscious client wants to use a USB > disk to sync his desktop to his laptop at home and has tried a number > of apps that do not work well. > > He was very keen on the commercial service Sugar-Sync until it started > to frag his data and required regular manual cleanups via their tech > service. He has since soured on Sugar. > > time machine seems to be a versioning app rather than a synchro app. > I haven't used it, but he has and claims that it doesn't do 2way > syncs. > > He's tried to use Bombich's Carbon Copy Cloner, but it does not (and > will not) allow 'remote to local' sync'ing bc they don't trust users > to avoid overwriting critical boot disk files. > > I set him up with an crontabbed rsync backup for regular sync's to a > remote server, which he's quite happy with, but he's not a CLI geek > and has already come close to wiping out his archives by playing with > the options, so I'm reluctant to set up such a system for him to do > this with naked rsync. > > I tried Unison for him (this is what it was designed to do) and while > the GUI is simple, it works very well and does allow both local 2-way > syncing (as to a USB disk) and remotely (via a number of protocols). > It also has a native Aqua interface, but there's no Snow Leper > finkable version. > > Are there any other obvious-to-a-Mac-user choices that I've missed? > > hjm > > > -- > Harry Mangalam - Research Computing, NACS, Rm 225 MSTB, UC Irvine > [ZOT 2225] / 92697 949 824-0084(o), 949 285-4487(c) > MSTB=Bldg 415 (G-5 on > --- > It is better to be roughly right than precisely wrong. > Keynes > _______________________________________________ > List-Info: https://maillists.uci.edu/mailman/listinfo/uccsc From ewieland at medicine.ucsf.edu Sat Nov 21 14:03:51 2009 From: ewieland at medicine.ucsf.edu (Wieland, Erik) Date: Sat, 21 Nov 2009 14:03:51 -0800 Subject: [UCCSC] Any other CrashPlan users out there? Message-ID: <24DF87DA-1711-41C2-BA1E-40A934742BA6@medicine.ucsf.edu> A lot of groups at UCSF are adopting CrashPlan Pro for backups. The company offers educational discounts of 25% on most products, but if we can come up with 5000 licenses across the system we qualify for even deeper discounts and maybe free personal licenses for students. We have close to 1000 at UCSF. Anyone else out there using CrashPlan? -- Erik Wieland Director of IT Services UCSF Department of Medicine Phone 415-502-7822 DOM Helpdesk 415-476-6827 [cid:3327505438_4288012] -------------- next part -------------- A non-text attachment was scrubbed... Name: image.jpg Type: image/jpeg Size: 2494 bytes Desc: image.jpg Url : http://maillists.uci.edu/mailman/public/uccsc/attachments/20091121/c45606cf/attachment.jpg From harry.mangalam at uci.edu Sun Nov 22 20:09:52 2009 From: harry.mangalam at uci.edu (Harry Mangalam) Date: Sun, 22 Nov 2009 20:09:52 -0800 Subject: [UCCSC] File Synchronizer for MacOS X In-Reply-To: <200911201424.39153.harry.mangalam@uci.edu> References: <200911201424.39153.harry.mangalam@uci.edu> Message-ID: <200911222009.52608.harry.mangalam@uci.edu> Thanks very much to all who replied. There's no clear winner for Snow Leopard, but I'm leaning towards Unison for the actual sync'ing process that my client requested, as long as I can get it compiled on a 10.6 machine. Here are the answers, compiled into one page. If you have additional info, I'd be happy to add it. hjm On Friday 20 November 2009 14:24:39 Harry Mangalam wrote: > Is there a good, free, OSS, or commercial file synchronizer for the > Mac? Or a verified web app that does the same? > > A (surprisingly) data-security-conscious client wants to use a USB > disk to sync his desktop to his laptop at home and has tried a > number of apps that do not work well. > > He was very keen on the commercial service Sugar-Sync until it > started to frag his data and required regular manual cleanups via > their tech service. He has since soured on Sugar. > > time machine seems to be a versioning app rather than a synchro > app. I haven't used it, but he has and claims that it doesn't do > 2way syncs. > > He's tried to use Bombich's Carbon Copy Cloner, but it does not > (and will not) allow 'remote to local' sync'ing bc they don't trust > users to avoid overwriting critical boot disk files. > > I set him up with an crontabbed rsync backup for regular sync's to > a remote server, which he's quite happy with, but he's not a CLI > geek and has already come close to wiping out his archives by > playing with the options, so I'm reluctant to set up such a system > for him to do this with naked rsync. > > I tried Unison for him (this is what it was designed to do) and > while the GUI is simple, it works very well and does allow both > local 2-way syncing (as to a USB disk) and remotely (via a number > of protocols). It also has a native Aqua interface, but there's no > Snow Leper finkable version. > > Are there any other obvious-to-a-Mac-user choices that I've missed? > > hjm -- Harry Mangalam - Research Computing, NACS, Rm 225 MSTB, UC Irvine [ZOT 2225] / 92697 949 824-0084(o), 949 285-4487(c) MSTB=Bldg 415 (G-5 on --- It is better to be roughly right than precisely wrong. Keynes