Amazon S3

Spent time over the weekend reading up on Amazon's new(ish) S3 (Simple Storage Service.)
Basically it's web based storage priced at $.15 / Gigabyte Month and $.20 / Gigabyte bandwidth. It's only accessible via their APIs (SOAP and REST) but a number of front ends have been created (including the S3Organizer Firefox extension.)

I think this as potential as storage and processing for the PACT project at work. I was able to create a proof of concept script that uploads files via http using the curl utility. I think this could replace the Autosubmit via ftp logic.

S3 combines very nicely with the new Amazon EC2 (Elastic Computing Cloud) service that just went into beta. It basically allows you to have on-demand computing (ala Sun Grid) for Linux systems. Storage is provided via S3 and computing is prices at $.10 / hour.

EC2 might be a solution for our data processing and analysis needs. We currently run some fairly heavy hitting (especially in memory use) R scripts to analyze the data and generate html based reports. The EC2 hosted systems could do the analysis and web serving.