Delivering dynamic non-HTTPS content over HTTPS with the help of AWS
Having a HTTPS-encrypted website is becoming more and more important. In fact, by now it should already be the standard and we should not even have to talk about this topic anymore.
As we started migrating our customers' websites to HTTPS, we faced the problem, that webcams most of the time weren't hosted on HTTPS-enabled domains and/or their providers had no plans to migrate them to HTTPS anytime soon.
But our customers of course wanted to have them on their websites. So we had to work out a solution, which could be realized fast and doesn't cause any maintenance costs. There are of course a few different ways to do this. But I'll explain the one we found the most useful in our case.
We have an AWS Lambda function, which fetches images from URLs stored inside an array and saves them to a S3 Bucket. This Lambda function is triggered every 10 minutes by a CloudWatch Event. To allow us to serve the images from a custom domain (with HTTPS-support), we set up CloudFront with TTL set to 0 (more on this below).
Our AWS Lambda function has the sole purpose of fetching all images from a URL and store them to an S3 bucket with a provided file name (to avoid duplicate names). For this purpose, we use Node.js' internal
Additionally to just fetching the image, we also do some pre-checks, if the URL is being redirected and follow those redirects to make sure we actually fetch the real image instead of the redirection page.
AWS CloudWatch Event rules allow us to trigger a Lambda function every 10 minutes, so we regularly get fresh images from the origin servers.
AWS S3 (Simple Storage Service) provides us with persistent and replicated storage for all our webcam images and simplifies the process of adding HTTPS to our webcams. It also keeps us distant from managing storage permissions etc.
You might wonder, why we had to set up CloudFront additionally to the S3 bucket. At the time of writing it was not possible to have a S3 bucket with a custom domain and a valid TLS-certificate, that's why we needed to set up CloudFront additionally to the S3 bucket.
Now there's usually a problem with CDNs when delivering dynamic content, as it's primary purpose is to deliver static content which doesn't change too frequently. But luckily, CloudFront has an option for this purpose.
AWS CloudFront can be set up with a TTL of 0 seconds, which means, that CloudFront will still cache the content, but it will make a GET request with an If-Modified-Since header to allow the origin to tell CloudFront, if the image can still be served from the cache or if it has to be pulled from the origin.
Using cloud services for such simple services can really save up time in terms of maintenance and management, as you don't have to care about OS updates, public-facing storage, distribution
Here's a list with a few resources you might check out to learn more about Amazon Web Services (AWS):
- AWS Lambda - https://aws.amazon.com/lambda/
- AWS CloudFront - https://aws.amazon.com/cloudfront/
- AWS S3 - https://aws.amazon.com/s3/
- AWS CloudWatch - https://aws.amazon.com/cloudwatch/
If you have any further questions, aspects you are unsure with or found anything, that is incorrect in this article, don't hesitate to leave a comment below or reach out to me on Twitter: @ivansieder.