Nipping at Dropbox's heels

There is a real resurgence of cloud storage taking place since Dropbox first launched some years ago and quickly rose above (and for the most part crushed) its competition at the time.  I'm going to highlight what I thought was going on earlier in the year, what's happened since then, and then highlight two new services.

Earlier this year I wrote about p2p file sharing as a threat to Dropbox, particularly the Cubby service.  The problem is, Cubby got their pricing model wrong (you have to pay for DirectSync, their p2p product) and haven't changed it.  Too bad for them, they'll never compete successfully with Dropbox in this way.  I don't expect them to disappear any time soon, but I do expect them to idle along as a bit player in the cloud storage space.  However, if they switched their Directsync feature into their "Basic" (freemium) package and focused on a pay model of central mirrored/subset storage for backup and faster syncs of critical data, it could still be a different story.  But probably not for much longer.

I also looked at BitTorrent Sync earlier this year.  I had various issues with it and while I like it, and ultimately lost trust in it due to regularly disappearing files from a folder I had under test between several machines.  I no longer use it although in the right circumstances I might still use it (e.g., mirroring a lot of content between multiple locations for a limited duration and easily verifiable results).  I continue to think that BT Sync will remain in the tech fringes for now - they will continue to be hampered by the need for a central server (is the server really secure?!), shared secrets approach (who can see my secrets and my files?!) and lack of open source (what am I really installing on my box?!)

Two interesting things have happened since then in the cloud storage world:

1. A new service called Copy has been making the rounds.  The good news is that you receive 20GB up front (if you activate your account through a referral link; otherwise you receive 15GB for direct signups).  This is 10x more space than Dropbox's upfront free 2GB of space.  I'm using Copy a lot now rather than Dropbox.  Copy has built their own infrastructure rather than using Amazon's AWS as Dropbox has done, I'm assuming that's given them much better cost structures and allows for this much greater level of freemium marketing. After a few weeks of testing, I switched to using it actively.  I've now been using it for a few months, no issues.  [Disclosure: If you click the links above you get 20GB rather than 15GB because its a referral and yes I get free space for the referral - thank you very much!] [Postnote: Warning! See comment below]
2. On the p2p storage side I stumbled on ownCloud.  I have it running on my own cloud IaaS server (Digital Ocean - working great and dirt cheap at $5 per month for a basic server instance) and have a few nodes connected to it.  It feels clunkier than Dropbox and Copy, but it does seem to work and most importantly, it's free! I'm testing it now and haven't switched to using it as a primary cloud file store yet.  [Note: I've only used at as network storage, not for Contacts and other features.]

Last thing to note - I've given up on BoxCryptor and moved on to using an OS X Extended/Journaled 256 bit AES, sparse bundle disk format on each of these cloud services.  I've had to give up multi-platform access that came with BoxCryptor but on the positive side I'm not paying a yearly subscription to BoxCryptor and I have filename encryption as well.  I'm currently testing volumes in Dropbox, Copy, Google Drive and ownCloud, so far so good with all of them. [Postnote: read comments for a pain point on this approach].

So that's how things look in August 2013 for best practices in cloud storage:

• Use Copy.  It works.  Much more storage for free.  Good for non-technical folks.  Or just keep using Dropbox if you have enough storage with them and/or don't mind paying for your storage.  [Postnote: Warning! See comment below]
• If you're technical enough, use ownCloud on your own server.  It looks viable for a do-it-yourself if that's important to you and it's opensource as well.
• If you're technical enough, value security, and don't need multi-platform access to your files, use an encrypted multi-file filesystem like OS X's sparse bundle format to store your folders and files [Postnote: see comments].

Pesky winmail.dat with Outlook, Apple Mac OS X Mail and me.com IMAP folders

I use me.com's IMAP folders to file email as part of an inbox zero policy I inflict on myself.  I recently had to resume using Exchange 2010 based email, but unfortunately not (yet!) directly connected to Apple Mac OS X Mail.  To access the Exchange mail, I've elected to use Outlook inside a Windows 7 VM.  I then hooked in my me.com IMAP folders into Windows Outlook so the Exchange and me.com folders are side by side in Outlook.  I then added an Outlook rule to copy new email that hits my Exchange Inbox to a me.com's Inbox.

This worked ok until I opened my first message in OS X Mail that had an attachment.  Instead of a normal attachment, I instead found a "winmail.dat" file.  Funny, I'd not seen one of those in a long time.

After some digging, I came across two ways to deal with the winmail.dat attachment problem:

1. The first and not recommended choice is TNEF's Enough.  The price is right at free/donations.  However, it's usability is awkward as you are required to open the *.dat file in a separate application that unpacks the TNEF format file from Exchange/Outlook and gives you the option to save the file.  I tried it, too painful.

2. The second and recommended choice is Letter Opener.  It is between USD 30-50.  It is seamlessly integrated with OS X Mail so you don't ever see a *.dat file, you see the attachments as you would in Outlook.

So if you stumble on this problem and need to consider TNEF's Enough versus Letter Opener, I'd recommend Letter Opener.

Enabling GNUPG (PGP) with Apple OS X mail.app

(Postnote 2011-03-05: Don't waste your time on the below.  Just go directly to gpgtools mail, read the instructions, and get on with it.  It's been updated to work with OS X 10.6 and Mail 4.4.  Just tested it, works great.)

I am so not an expert on PGP, GNUPG (GNU Privacy Guard) or OS X's mail.app.  But what I can do is explain how I got the basics of PGP working with Mac mail and some resources that helped.

If you don't know anything about PGP or want more detail, see "Learn More" section at the end of this post.

The following worked for Mac OS X 10.6.3 and mail.app 4.2.

1. Install GNU's Privacy Guard (gnupg).

You need to have Macports installed.  Install it if you don't have it.

sudo port install gnupg

2. Generate your encryption key.

gpg --gen-key

Here are the options I used:

1. Option 2: DSA and Elgamal
2. Keysize: 3072 (that was the biggest keyvalue offered)
3. 0, key does not expire
4. Key identification
Real name: Jeff Blogs
email address: jeffblogs@dodgymail.com
No comment
5. Passphrase "something memorable yet complicated and long, don't share it with anyone, and don't forget it"


Your ~/.gnupg directory of configuration and databases gets set up.

3. Install the magic mail.app bundle

The bundle contains a version of GPGMail that works with OS X 10.6.3.

Exit mail.app.

mkdir ~/Library/Mail/Bundles  # if it doesn't exist already - mine didn't

Be thankful for clever, helpful and giving people and Download the bundle.

Extract from zip download and deposit GPGMail.mailbundle into ~/Library/Mail/Bundles

From the command line as the user you run mail with (not root!):

defaults write com.apple.mail EnableBundles -bool true
defaults write com.apple.mail BundleCompatibilityVersion 3

Start mail.app.

You should now have a PGP option in your mail menu (Message->PGP).

Mail.app menu with new PGP option

You should also see a PGP toolbar when you create a new email:

New PGP toolbar appears when composing a new email

(This step was the silver bullet from macrumors.com forum with an updated GPGMail from Lukas Pitschl - thank you!)

4. Create your public key.

From command line:

gpg --armor --output "Jeff Blogs.asc" --export jeffblogs@dodgymail.com

You'll need to send people your public key if you want them to send encrypted email back to you.

5. Add other people's public keys

gpg --import "Ronald McDonald.asc"

At this point you should now be able to send and receive PGP encrypted emails and mail.app will be reasonably supportive of you.

I found regularly restarting mail.app is useful when fiddling with gpg at the command line.

6. Set yourself up with a verified key service.  This will decrease warnings from mail and GNUPG.

Set yourself up with pgp.com.

Use the name and email address you used to generate your key in step 2 above.

Add the verified key service key:
gpg --import keyserver1.pgp.comGlobalDirectoryKey.asc

Let GNUPG know about the pgp.com key server.  Edit ~/.gnupg/gpg.conf and uncomment "keyserver ldap://keyserver.pgp.com" line.

(You're restarting mail.app between these steps right?)

7. Learn more!

These were helpful to the above:

• knuthbert.com - How to use GPGMail with Mac OS X 10.6 (Snow Leopard)
• sente.ch - PGP for Apple's Mail

These might have been helpful if they weren't really long, complicated, out of date, didn't work and I didn't already have the basic idea of how PGP was supposed to work:

• linuxmafia.com - An overview of GNUPG and PGP brief
• gnupg.org - The information mothership, good luck

And of course GPGMail itself, which doesn't work with current versions of Snow Leopard and mail.app.

-----

2010-06-19 Postnote: The latest OS X upgrade to Mail 4.3 disabled gpgmail.  Two things to fix this:

1. Copy GPGMail.mailbundle from "~/Library/Mail/Bundles (Disabled)" to ~/Library/Mail/Bundles

2. Enter the GPGMail.mailbundle directory and add two new UUIDs to Info.plist in the "SupportedPluginCompatibilityUUIDs" section:


E71BD599-351A-42C5-9B63-EA5C47F7CE8E
B842F7D0-4D81-4DDF-A672-129CA5B32D57


And gpgmail is working again.

(As outlined by user Bytes_U on the Apple support forums.)

Using MobileMe's iDisk as an interim backup while traveling

Introduction

I use an Apple laptop hard disk as my primary (master) data storage device.  To provide interim backups while traveling, I use Apple's MobileMe iDisk for network backups to supplement primary backups only available to me when I'm at home.

Having dabbled with iDisk for a few years, I have two key constraints for using iDisk:

• I don't always have a lot of bandwidth available (e.g., a mobile phone GPRS connection) and I don't want a frequent automatic sync to hog a limited connection.
• I don't trust MobileMe with primary ownership of data or files.  Several years ago I switched to using the iDisk Documents folder (with local cache) for primary storage but then had several files magically disappear.

I've now evolved to using iDisk as a secondary backup medium.  I manually run these steps when I have plenty of bandwidth available.  There are two steps to this:

• rsync files/folders from specific primary locations to a named directory under iDisk
• Sync the iDisk

How to do it

The rsync command I use looks like this:

for fn in Desktop dev Documents Sites; do

   du -sk "/Users/my_username/$fn" | tee -a ~/logs/laptop_name-idisk.rsync.log

   rsync -avE --stats --delete "/Users/my_username/$fn" "/Volumes/my_mobileme_name/laptop_name/Users/my_username" | tee -a ~/logs/laptop_name-idisk.rsync.log

done

The rsync flags in use:

-a         archive (-rlptgoD no -H)

           -r    recursive

           -l    copy symlinks as symlinks

           -p    preserve permissions

           -t    preserve times

           -g    preserve group

           -o    preserve owner

           -D    same as "--devices --specials" (preserve device and special files)

-v         verbose

-E         preserve extended attributes

--stats    detailed info on sync

--delete   remove destination files not in source

Explanation:

• I'm targeting specific locations that I want to backup that aren't overly big but tend to change frequently (in this case several folders from my home directory: Desktop, dev, Documents, Sites)
• A basic log is maintained, including the size of what is being backed up (the "du" command)
• I use rsync rather than copy because rsync is quite efficient - it generally only copies the differences, not the whole fileset.
• The naming approach on the iDisk allows me to keep a backup by laptop name allowing me to keep discrete backup collections over time.  My old laptop and backups sit beside my current laptop backups.
• The naming approach also means I don't use any of the default directories supplied by iDisk as I'm not confident that Apple won't monkey with them.
• ~/Library/Mail is a high change area but not backed up here (see below for why)

The rsync updates the local iDisk cache.  Once the rsync is complete (after the first rsync I find it takes less than 10 seconds for subsequent rsyncs), manually kick off an iDisk network sync (e.g., via a Finder window, clicking on the icon next to iDisk).

An additional benefit to having a network backup of my important files and folders is that I can view and/or edit these files from the web, iphone, or PC.  I find that being able to access email/IMAP from alternative locations is the most useful feature, but I have had minor benefit from accessing files as well when my laptop was unavailable or inconvenient to access (e.g., quick check of a contract term in the back of a taxi on an iphone).

Other Backups

I have two other forms of backups:

• Irregular use of Time Machine to a Time Capsule, typically once a week if my travel schedule permits.
• MobileMe's IMAP for all email filing (and IMAP generally for all email).

Basically, if I'm traveling, I rely on rsync/iDisk and IMAP for backups.  I also have the ability to recover a whole machine from a fairly recent Time Machine backup.

Success Story

In June 2009 I lost my laptop HDD on a return flight home after 2 weeks of travel.  I had a Time Machine backup from right before I'd left on travel, and occasional iDisk rsyncs while traveling.

Once I got home I found an older HDD of sufficient size and restored from the Time Machine image from the Time Capsule.  This gave me a system that was just over 2 weeks "behind".  Once IMAP synchronized my mailboxes, that only left a few documents missing that I'd created while traveling.  Luckily I'd run an rsync and iDisk right before my return flight, so once I'd restored those, I'd recovered everything I'd worked on over the two weeks of travel, only missing only some IMAP filing I'd done on the plane.

Weakness

The primary flaw in my approach is that you have to have the discipline to remember to manually kick off the rsync and iDisk sync after you've made changes you don't want to lose.  I certainly don't always remember to run it, nor do I always have a good Internet connection available to enable it.  However, I find that remembering sometimes is always better than not having any recent backup at all.

Alternative Approaches

An obvious alternative is to use the MobileMeBackup program that is preloaded onto your iDisk under the Software/Backup directory.  Using this tool, you should be able to perform a similar type of backup to what I've done here.  I've not tried it as it was considered buggy back when I first started using iDisk for network backups.  I'll likely eventually try this and may shift to it if it works.

A viable alternative approach is to carry around a portable external hard drive, and make Time Machine backups to it more frequently than you would otherwise do over the network via iDisk.  You could basically keep a complete system image relatively up-to-date if you do this.  More hassle, but lower risk and easier recovery if your primary HDD fails.  However, if you get your laptop bag and external HDD stolen, you'll be worse off.

While on holiday recently, I was clearing images off of camera SD card memory as it filled up.  I put these images both on the laptop HDD and an external HDD.  This protects me from laptop HDD failure, but wouldn't help if both the laptop and external HDD was stolen.

iDisk Comparison to DropBox

DropBox is a popular alternative to iDisk.  I find DropBox to be better at quickly and selectively sharing files, it has better cross-platform support (particularly with a basic Android client), and it's sync algorithm seems to work better than the iDisk equivalent.  You could certainly do everything described here with DropBox.

The downside with DropBox is having to pay $120 per year for 50GB of storage versus $60-100 per year ($60 on promotion, e.g., with a new Apple laptop; otherwise $100) for 20GB of storage with MobileMe.  I find 20GB to be plenty for IMAP, iDisk and photos providing I filter out big auto-generated emailed business reports (store on laptop disk not in IMAP), and only upload small edited sets of photos.  I'll probably exhaust the 20GB in 2-3 more years at my current pace, but I'd expect Apple to increase the minimum by the time I would otherwise be running out of space.

MobileMe is of course more than just iDisk, so if you use more of it's features, it increases in value relative to DropBox.

Both iDisk and DropBox are usable choices, the differences are not sufficiently material to strongly argue for one or the other.  I have seen iDisk improve over the last few years and I'd expect Apple to eventually catch up with DropBox.

Conclusion

While I'm not confident in using MobileMe's iDisk as a primary storage location, I have found it useful as a network backup.  Combined with normal backups using Time Machine and Time Capsule, it provides a high-confidence recovery from damaged or lost primary use laptops.