The Turnstone's Bill

Rsync and S3 With DropSync

Amazon s3 is a reliable and relatively cost effective way to backup data offsite. Although s3 isn’t a filesystem, Amazon provide an API for accessing s3 data which has allowed developers to create s3fs, a program which presents s3 data as if it existed as files on a network drive.

This means that in theory it is now possible to backup data directly to s3 using DropSync. At this stage though the setup is not for the faint of heart … and there are some serious caveats (see below).

Setting up s3fs

  1. Install xcode and xcode commandline tools
  2. Install Homebrew

    ruby <(curl -fsSk https://raw.github.com/mxcl/homebrew/go)
    
  3. Install s3fs

    brew install s3fs
    

    Then follow instructions from

    brew info fuse4x-kext
    
  4. Mount your bucket with s3fs

    echo "accessKeyId:secretAccessKey" > ~/.passwd-s3fs  
    chmod 600 ~/.passwd-s3fs  
    mkdir -p /path/to/mounts3  
    s3fs bucketname /path/to/mounts3
    
  5. Now your bucket should appear as an accessible shared drive. You can now use dropsync to access this as normal

Caveats

Even though s3fs does a good job of presenting s3 to you as a normal set of files on your filesystem it cannot escape the fact that s3 is not like a normal filesystem. Several issues come up;

  1. S3 has no inherent concept of directories. This means that different programs accessing s3 adopt different conventions about how to encode directory structure. s3fs uses a different convention from amazon’s management console which means that in most cases you will only see directories if they were created by s3fs itself.
  2. S3 is a distributed filesystem and can only guarantee “eventual consistency” which means that its entirely possible to write some data to your s3 bucket only to find that when you read it moments later you don’t see that data. If you wait long enough it should reappear.
  3. Network latency was a big problem for me. This might be because I’m in Australia and we don’t have any Amazon datacenters particularly close ( I think the closest is Singapore)