Wednesday, March 28, 2007

Amazon S3 with cURL?

As I've talked about in previous posts, I've been searching for a good way to access Amazon S3 from Suneido (to use for backing up our customer's databases).

The SmugMug presentation recommended using cURL. We already use cURL for a variety of tasks (such as FTP) so we'd be happy to use this option. But S3 requires SHA-1 hashes which I didn't think cURL could do. Maybe you can calculate the hashes separately and use cURL just for the transfer. I'll have to look into this.


Irving Reid said...

By SH1, do you mean SHA-1? For that, and pretty well any other cryptography you might want to do in a C or C++ program, you'll want the OpenSSL library. I've used it for years. It can also be integrated with cURL to support https.

andrew said...

Oops, yes, I meant SHA-1.

We do use the version of cURL with OpenSSL and I had found other references to OpenSSL.

There's also a GNU standalone sha1sum utility we considered using. (GnuWin32 has a Windows version)

The next trick is to get the right headers in the right form to hash. Not rocket science but a little tricky. And you have to make sure the headers you hash are identical to the headers that cURL sends. It's obviously possible since there's a Perl script (s3curl) that uses cURL to talk to S3.

We ended up using Sharp3hell for now since it already did everything we needed. Maybe down the road we'll work out the details to do it ourselves.