I recently spent about a week working on a prototype for a new product idea. Another engineer and I needed to put something together quickly, so we could get feedback from potential customers that will inform the product. We already host our production servers on AWS, so I looked at the AWS console to see what we could use to get our prototype up and running fast. We created a mobile/desktop application that allows uploading photos, automatically resizing and extracting metadata, and displaying and interacting with the photos on a map. We did this in just a few days by knitting together a number of AWS building blocks, including the relatively new Lambda service. At demo time, it looked like this:
I needed to make a prototype for use in customer demos. We need to be able to take pictures with a phone, upload them, extract metadata, and view them on a desktop. Now what?
Somewhere to put photos
This one is easy: I created an S3 bucket. Since this is just a prototype, I wanted to spend as little time as possible on user authenticated. I set permissions on the bucket so that everyone can upload and download content. Create bucket, check some boxes on the properties; done.
Photo thumbnails
Now we can upload photos through the AWS console or command line to the bucket, but the photos are way too big to use on a website. The newly-released AWS Lambda product is perfect for this, especially since the Lambda walkthrough has an example for exactly this. I cut and pasted the code, modified it a bit for my purposes, and uploaded it to Lambda. Next, I configured my content bucket to run the resize function whenever an image is uploaded: Events / Add Notification / Object Created, Lambda function
Photo metadata
The photos contain metadata that we want to extract and save for later. I installed exiftool to examine photo metadata on the command line and find the names of interesting fields.
An AWS Lambda function can include any node modules, so a quick search led me to node-exif. I added another steps to the waterfall in my image processing Lambda function to pull out date and location metadata:
function loadImage(response, next) {
new ExifImage({ image: imageBody }, next);
},
function getMetadata(exifData, next) {
if (exifData.gps) {
var lat = exifData.gps.GPSLatitude;
var lng = exifData.gps.GPSLongitude;
if (lat && lat.length >= 3 && lng && lng.length >= 3) {
item.latLng = {'NS': [
degreesToDecimal(lat[0], lat[1], lat[2], exifData.gps.GPSLatitudeRef)+'',
degreesToDecimal(lng[0], lng[1], lng[2], exifData.gps.GPSLongitudeRef)+''
]};
...
}
Metadata storage
Once I got the metadata out, I need to save it somewhere. Although our production systems use MongoDB and Postgres, they’re inside our VPC and are hooked into the rest of our systems for permissions, access, etc. I wanted something quick and easy to use with my new bucket and Lambda function. I saw DynamoDB on the AWS console, and a few clicks later, I had a database ready to save the metadata. To hook it up my Lambda function, I just needed to add DynamoDB permissions to the role that the Lambda function uses:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1431450195000",
"Effect": "Allow",
"Action": [
"dynamodb:DeleteItem",
"dynamodb:GetItem",
"dynamodb:GetRecords",
"dynamodb:PutItem",
"dynamodb:Query",
"dynamodb:Scan",
"dynamodb:UpdateItem"
],
"Resource": [
"arn:aws:dynamodb:us-east-1:891208296108:table/PrototypeContentBucket"
]
}
]
}
Then I updated my Lambda script to save the metadata directly to DynamoDB using the AWS Javascript API:
var params = {
Key: { 'key': { 'S': srcKey } },
TableName: 'PrototypeContentTable',
};
var expressions = [];
var index = 0;
var values = {};
for (var attr in item) {
expressions.push(attr+'=:v'+index);
values[':v'+index] = item[attr];
index += 1;
}
params.UpdateExpression = 'SET '+expressions.join(', ');
params.ExpressionAttributeValues = values;
dynamodb.updateItem(params, next);
Access to the database
With one script and some configuration, I now had a system that makes a thumbnail, extracts metadata, and saves it to a database whenever an image is uploaded to an S3 bucket. Next, I needed access to the data. I started up a local web server with python -m SimpleHTTPServer, and looked at how to talk to the database. AWS doesn’t care that my database is just for testing, and I need to prove that I should be allowed to access it.
AWS allows authenticated via other services, and has step-by-step directions for authenticated with Google. I created Google app, created role that allows getting and putting to content and thumbnails buckets and DynamoDB, and cut and paste the Google button into my static html page.
When my app hears back from Google, it sets up the connection to AWS services:
// Google auth callback
function signinCallback(authResult) {
if (authResult['status']['signed_in']) {
$('#signinButton').hide();
setupAWSCredentials(authResult.id_token);
}
}
function setupAWSCredentials(token) {
AWS.config.credentials = new AWS.WebIdentityCredentials({
RoleArn: 'arn:aws:iam::AWS_account_id:role/Prototype_GoogleAuth',
WebIdentityToken: token
});
s3 = new AWS.S3();
dynamodb = new AWS.DynamoDB();
authenticated();
}
Data to the browser
Now that the browser is allowed to talk to my S3 bucket and DynamoDB, I used the AWS SDK for the browser to load my metadata records into the browser. It’s a prototype, so I just fetch all of the records.
dynamodb.scan(params, function(error, data) {
if (error) {
console.log(error, error.stack); // an error occurred
} else {
data.Items.forEach(function(item) {
....
}
}
}
Website
So far, the UI for my prototype was only available on localhost. I needed a public URL so that my co-workers could see it too. While we have lots of webservers, they each have their own codebases and deploy processes, and didn’t want to try to bolt on short-lived prototype code to a real system. But setting up a whole new webapp didn’t sound like a good idea either. Instead, I took advantage of static hosting from an S3 bucket, and I had a webserver in just a few clicks. I created a bucket, uploaded my html and JavsScript, and when to bucket Properties / Static Website Hosting / Enable website hosting. This gave me a URL like http://prototype-ui.s3-website-us-east-1.amazonaws.com, where a user could sign in with Google, and load metadata from DynamoDB. It didn’t actually do anything with the data, but that’s exactly where I wanted to spend my prototyping time.
Done
After writing a small amount of code and clicking through some configuration screens, I had an basic application that actually did quite a lot:
- authenticated users with a Google account
- uploaded photos to the the cloud
- created a thumbnail version of each photo
- extracted GPS and other metadata and saved it to a database
- served a website that allows browsing and updating photo metadata on a map
Using AWS allowed me to concentrate on the parts I wanted to demo (pretty maps!), and get it working in just a couple of days. While the product we build will likely look pretty different, having a working prototype to show will help us get valuable feedback to help set our course. It was also a quick, low-risk way to try out some of the service offerings from AWS that we hadn’t used before (Lambda? wow, what else can we use it for?! DynamoDB? let’s stick with Postgres).
