Amazon has announced a new service - SimpleDB
We are pretty happy with our use of Amazon's S3 (Simple Storage Service)
I've been curious to try Amazon EC2 (Elastic Compute Cloud) but I haven't found a good application yet.
One of the big limitations with EC2 is that it's not well suited to running database servers. I've been waiting for them to improve support for this, but instead (or at least, first) we get SimpleDB.
I wonder if someone will make Rails work with SimpleDB as the database? How would the performance compare?
Showing posts with label s3. Show all posts
Showing posts with label s3. Show all posts
Friday, October 19, 2007
Using S3 for Customer Backups
For some time we've been using S3 to back up our customers systems.
Originally we set them up to FTP the data back to our office. This was handy for us because if we wanted to look at their data we had it in-house. But as the number of clients and the size of their data grew we ran out of bandwidth. (Thankfully we did these transfers at night so it didn't slow down our daytime internet access.)
The other problem was the sheer size of the data and the issues of how to guard against disk failure. Although we don't promote this to our clients as a "backup" service many of them still end up falling back on us when they discover their own backups weren't done or were no good. And people often don't discover problems till days or weeks later so they need older copies, not just the most recent.
We decided to give S3 a try and so far it has worked out well. We're up to about 8 gb of data transfer per night and we currently have about 240 gb of storage. This is costing us about $50 per month - a bargain as far as I'm concerned.
We are currently keeping the last 8 days, last 5 weeks, last 13 months, and every year, or something like 25 - 30 copies per customer. On top of this redundancy, Amazon says they store multiple copies of each file.
A minor downside is that when we want to look at someone's data we have to download it, but that's not a big deal. And it's still a lot easier to download from S3 than to download directly from the customer.
There is a potential concern with storing data with a third party, but we encrypt the files and Amazon has decent security on top of that, so it seems ok. It's doesn't seem any worse than other hosting situations.
Overall, we're pretty happy with this setup.
Originally we set them up to FTP the data back to our office. This was handy for us because if we wanted to look at their data we had it in-house. But as the number of clients and the size of their data grew we ran out of bandwidth. (Thankfully we did these transfers at night so it didn't slow down our daytime internet access.)
The other problem was the sheer size of the data and the issues of how to guard against disk failure. Although we don't promote this to our clients as a "backup" service many of them still end up falling back on us when they discover their own backups weren't done or were no good. And people often don't discover problems till days or weeks later so they need older copies, not just the most recent.
We decided to give S3 a try and so far it has worked out well. We're up to about 8 gb of data transfer per night and we currently have about 240 gb of storage. This is costing us about $50 per month - a bargain as far as I'm concerned.
We are currently keeping the last 8 days, last 5 weeks, last 13 months, and every year, or something like 25 - 30 copies per customer. On top of this redundancy, Amazon says they store multiple copies of each file.
A minor downside is that when we want to look at someone's data we have to download it, but that's not a big deal. And it's still a lot easier to download from S3 than to download directly from the customer.
There is a potential concern with storing data with a third party, but we encrypt the files and Amazon has decent security on top of that, so it seems ok. It's doesn't seem any worse than other hosting situations.
Overall, we're pretty happy with this setup.
Labels:
s3
G.ho.st - Global Hosted Operating System
G.ho.st – Home Page
Apparently this is implemented using Amazon S3 and EC2. Interesting idea, but pretty annoying how they maximize my browser window and hide the address and tool bars.
Apparently this is implemented using Amazon S3 and EC2. Interesting idea, but pretty annoying how they maximize my browser window and hide the address and tool bars.
Labels:
s3
Wednesday, March 28, 2007
Amazon S3 with cURL?
As I've talked about in previous posts, I've been searching for a good way to access Amazon S3 from Suneido (to use for backing up our customer's databases).
The SmugMug presentation recommended using cURL. We already use cURL for a variety of tasks (such as FTP) so we'd be happy to use this option. But S3 requires SHA-1 hashes which I didn't think cURL could do. Maybe you can calculate the hashes separately and use cURL just for the transfer. I'll have to look into this.
The SmugMug presentation recommended using cURL. We already use cURL for a variety of tasks (such as FTP) so we'd be happy to use this option. But S3 requires SHA-1 hashes which I didn't think cURL could do. Maybe you can calculate the hashes separately and use cURL just for the transfer. I'll have to look into this.
Labels:
s3
Day 3 at ETech 2007
Another good day at ETech. I've found a nearby Starbucks that is in a lovely courtyard and isn't very busy. This is where I start my day with a Grande Latte and a little peace before the mind-storm.
The keynotes were all good, but I really enjoyed Danah Boyd's presentation. It's hard not to appreciate someone who is so obviously passionate about their work. I also get a kick out of geeks who are brave enough to dress idiosyncratically. Personally, I can't imagine intentionally doing something to make myself stand out more! Except in my work, of course :-)
Lunch was outside in the Seaport Courtyard again. The weather was wonderfully sunny and warm (especially compared to yesterdays horrendous winds). After lunch I sat by the pond in Seaport Village for a quiet coffee before heading back for afternoon sessions. The sun was shining on the water and the ducks were entertaining. Mama duck brought her three ducklings within inches of my feet. I wished I had my camera.
The presentation on SmugMug's use of S3 was good. This year a number of the sessions have been thinly disguised marketing pitches, but this was a good, seemingly honest, first hand experience report.
Next was a combined presentation on Google's MapReduce and Hadoop - an open source implementation of MapReduce that is part of the Apache Lucene search engine project. Although map and reduce are familiar from functional programming, their application to processing large data sets with clusters of computers is pretty neat. In the past, this would have been out of reach of most of us who don't have access to clusters, but now you can "rent" as big a cluster as you want from Amazon EC2 (or you will be able to once EC2 is publicly available).
The last two sessions of the day that I attended were from Microsoft and IBM on various research projects. One of the Microsoft projects was Baku a "visual" programming environment for kids. I wonder if some of the ideas could be applied to allow end-users to do more complex "programming" in applications. One of the IBM projects was Koala - basically a macro recorder for the web with a unique slant towards sharing the resulting scripts on a wiki. Unfortuately, neither of these projects is publicly available yet.
I had supper at the Edgewater Grill in Seaport Village. The food and service were nothing great, but I enjoyed watching the sunset over the bay.
After supper I stopped in at the MAKE Fest but they only had a handful of exhibits so I didn't stay long.
The keynotes were all good, but I really enjoyed Danah Boyd's presentation. It's hard not to appreciate someone who is so obviously passionate about their work. I also get a kick out of geeks who are brave enough to dress idiosyncratically. Personally, I can't imagine intentionally doing something to make myself stand out more! Except in my work, of course :-)
Lunch was outside in the Seaport Courtyard again. The weather was wonderfully sunny and warm (especially compared to yesterdays horrendous winds). After lunch I sat by the pond in Seaport Village for a quiet coffee before heading back for afternoon sessions. The sun was shining on the water and the ducks were entertaining. Mama duck brought her three ducklings within inches of my feet. I wished I had my camera.
The presentation on SmugMug's use of S3 was good. This year a number of the sessions have been thinly disguised marketing pitches, but this was a good, seemingly honest, first hand experience report.
Next was a combined presentation on Google's MapReduce and Hadoop - an open source implementation of MapReduce that is part of the Apache Lucene search engine project. Although map and reduce are familiar from functional programming, their application to processing large data sets with clusters of computers is pretty neat. In the past, this would have been out of reach of most of us who don't have access to clusters, but now you can "rent" as big a cluster as you want from Amazon EC2 (or you will be able to once EC2 is publicly available).
The last two sessions of the day that I attended were from Microsoft and IBM on various research projects. One of the Microsoft projects was Baku a "visual" programming environment for kids. I wonder if some of the ideas could be applied to allow end-users to do more complex "programming" in applications. One of the IBM projects was Koala - basically a macro recorder for the web with a unique slant towards sharing the resulting scripts on a wiki. Unfortuately, neither of these projects is publicly available yet.
I had supper at the Edgewater Grill in Seaport Village. The food and service were nothing great, but I enjoyed watching the sunset over the bay.
After supper I stopped in at the MAKE Fest but they only had a handful of exhibits so I didn't stay long.
Tuesday, January 16, 2007
Amazon S3 Tools continued
Just to confuse things even more, I tried #Sh3ll from home on a big file and it worked fine (even over wireless).
So that would seem to indicate that the problem is with my network connection at work. Although that still doesn't explain why jSh3ll worked.
However, I didn't get around to trying the others and I only did the one test at home. So it could have just been lucky.
I'm not sure where that leaves me. Maybe we should try #Sh3ll (because it appears to be the easiest to deploy) on a few of our clients and see what happens.
So that would seem to indicate that the problem is with my network connection at work. Although that still doesn't explain why jSh3ll worked.
However, I didn't get around to trying the others and I only did the one test at home. So it could have just been lucky.
I'm not sure where that leaves me. Maybe we should try #Sh3ll (because it appears to be the easiest to deploy) on a few of our clients and see what happens.
Labels:
s3
Monday, January 15, 2007
Amazon S3 Tools
We ftp our client's database's back to our office nightly. This gives them an off site backup and it gives us quick access to their data to help support them. The problem is that as our client base has grown we have run into bandwidth limits at our office.
We could use a hosted ftp site, but I thought it might be worth trying Amazon's Simple Storage Service (S3).
The catch is that authentication is done with HMAC-SHA1 which we've never implemented for Suneido. So instead of writing our own interface to S3 I figured we could simply call someone else's command line tool.
There are quite a few available tools for S3. Here are some that I tried:
s3cmd - Ruby
#Sh3ll - C#
jSh3ll - Java
s3tools - Python
s3curl - Perl
I wanted something that would be easy to deploy on our clients, preferably without installing a language run-time. The best for this appeared to be #Sh3ll which was an exe that only required .Net 1.1 which should be installed on most recent Windows machines. However, there are tools to turn Perl, Python, or Ruby programs into standalone executables so that might be a deployment solution.
But the only one I could get to work consistently for large files was jSh3ll which requires Java 5. All the other tools give timeout errors. I'm not sure what jSh3ll is doing different, or whether Java just has better network support.
The question is whether it's practical to require the Java runtime to be installed on our customers servers. It doesn't seem unreasonable to me, but some people are touchy about it.
PS. There is a nice Firefox extension for S3
We could use a hosted ftp site, but I thought it might be worth trying Amazon's Simple Storage Service (S3).
The catch is that authentication is done with HMAC-SHA1 which we've never implemented for Suneido. So instead of writing our own interface to S3 I figured we could simply call someone else's command line tool.
There are quite a few available tools for S3. Here are some that I tried:
s3cmd - Ruby
#Sh3ll - C#
jSh3ll - Java
s3tools - Python
s3curl - Perl
I wanted something that would be easy to deploy on our clients, preferably without installing a language run-time. The best for this appeared to be #Sh3ll which was an exe that only required .Net 1.1 which should be installed on most recent Windows machines. However, there are tools to turn Perl, Python, or Ruby programs into standalone executables so that might be a deployment solution.
But the only one I could get to work consistently for large files was jSh3ll which requires Java 5. All the other tools give timeout errors. I'm not sure what jSh3ll is doing different, or whether Java just has better network support.
The question is whether it's practical to require the Java runtime to be installed on our customers servers. It doesn't seem unreasonable to me, but some people are touchy about it.
PS. There is a nice Firefox extension for S3
Labels:
s3
Subscribe to:
Posts (Atom)