Amazon s3 is a reliable and relatively cost effective way to backup data offsite. Although s3 isn’t a filesystem, Amazon provide an API for accessing s3 data which has allowed developers to create s3fs, a program which presents s3 data as if it existed as files on a network drive.
This means that in theory it is now possible to backup data directly to s3 using DropSync. At this stage though the setup is not for the faint of heart … and there are some serious caveats (see below).
Setting up s3fs
- Install xcode and xcode commandline tools
ruby <(curl -fsSk https://raw.github.com/mxcl/homebrew/go)
brew install s3fs
Then follow instructions from
brew info fuse4x-kext
Mount your bucket with s3fs
echo "accessKeyId:secretAccessKey" > ~/.passwd-s3fs chmod 600 ~/.passwd-s3fs mkdir -p /path/to/mounts3 s3fs bucketname /path/to/mounts3
Now your bucket should appear as an accessible shared drive. You can now use dropsync to access this as normal
Even though s3fs does a good job of presenting s3 to you as a normal set of files on your filesystem it cannot escape the fact that s3 is not like a normal filesystem. Several issues come up;
- S3 has no inherent concept of directories. This means that different programs accessing s3 adopt different conventions about how to encode directory structure. s3fs uses a different convention from amazon’s management console which means that in most cases you will only see directories if they were created by s3fs itself.
- S3 is a distributed filesystem and can only guarantee “eventual consistency” which means that its entirely possible to write some data to your s3 bucket only to find that when you read it moments later you don’t see that data. If you wait long enough it should reappear.
- Network latency was a big problem for me. This might be because I’m in Australia and we don’t have any Amazon datacenters particularly close ( I think the closest is Singapore)