Encrypted Full Email Storage / Backup API for AWS S3, Google Cloud Storage, Azure Blob

CloudMailin has allowed you to receive email via webhook for a number of years. One of the great features of CloudMailin is the ability to upload your email message attachments directly to Cloud Storage such as Amazon Web Services (AWS) S3, Microsoft Azure Blob Storage or Google Cloud Storage.

Offloading email attachments to Cloud Storage allows your Web Application / Webhook to receive a much lighter HTTP POST with only the email body and metadata without having to process large files and attachments.

Announcing CloudMailin Full Email Storage

We're excited to announce that customers are now able to upload the entire email directly to AWS S3 and have the URL of the upload delivered directly to your existing webhook. The full email can be uploaded as either the entire RAW email (as an EML file) or as a fully parsed email in JSON format.

Of course, as with uploaded email attachments, the email will be stored securely and encrypted within your own AWS S3 bucket, so long as your S3 bucket has encryption enabled.

There are a number of options that this feature opens up and we're really happy to be able to share it with customers. You could use it to asynchronously process inbound emails or even create a complete email backup solution stored in AWS. We think the ability to access entire email as JSON will be especially useful for building serverless functions. We're excited to see how our customers use the API.

How can I upload the entire Email to AWS S3

We're now accepting customers into the beta for full email upload to AWS S3. Simply contact us to find out more. We'll be rolling out the feature to all of our customers on the professional plan and above over the next couple of weeks.

We've been testing the feature ourselves for several months now and we're really pleased with the options and functionality that it opens up for us.

Can I send email to Azure Blob Storage or Google Cloud Storage?

Customers that want to upload to Azure Blob Storage or Google Cloud Storage can also apply to join the beta of this product, which should be available in a couple of weeks time.

Getting Set Up

There are a few simple steps to get setup sending email to Amazon Web Services S3. The process is very similar for Azure blog storage or Google Cloud Storage. However, we'll only cover AWS here.

First we need to set permissions to your AWS S3 bucket.
With permissions set we can enable full email storage in CloudMailin.

Give CloudMailin permission to your AWS S3 Bucket

There are several ways that you can grant permission for CloudMailin to create files in your S3 Bucket. The simplest is to use the Amazon Web Services (AWS) web console.

Head to https://console.aws.amazon.com and Sign in to the management console.
Click on the S3 Tab, then select your bucket (or create one)
Click the permissions button
Scroll down to the Access control list (ACL) section
Click Edit and then Add grantee
In the grantee field from above, paste the CloudMailin Canonical ID, choose write and then save

CloudMailin should now have permission to add files to your S3 bucket.

Send the full email to AWS S3 as EML or JSON

Now that you've added the permission to upload emails, head to CloudMailin and enable the feature.

Log into your Address Dashboard and choose the address you wish to manage.
Select Add full message storage.
Enter your AWS S3 bucket name and an optional path prefix to place before each file name (in the format bucket_name/path_prefix). You'll also need to add your S3 bucket region.
You will need to select the permission to apply to files when they are created. We recommend using Bucket owner full control here.
Finally, you'll need to select the Format.

What format are the emails stored in?

There are two formats available for the email:

RAW - This is the full raw email in EML format. This includes all of the headers and the body of the email but no parsing will have been performed on the email.
JSON - The email is parsed into JSON so that it can be more easily read. This makes working with the email much more simple. This will also contain any links to the attachments (fully parsed and stored) if you have email attachment cloud storage enabled.

What does email storage look like in the API?

Once you've enabled the full email cloud storage functionality, each HTTP POST to your webhook will contain a store_url parameter in the envelope. In the JSON format this will look similar to the following:

"envelope": {
  "to": "to@example.com",
  "from": "from@example.com",
  "helo_domain": "localhost",
  // ...
  "store_url": "https://example.s3.amazonaws.com/2021_12_01_00_00_00_a8ed6a3f7aeb93ce726b9dd2dea9eb53.eml"
}

It's important to note that, like the email attachment storage, the store_url parameter containing the URL to the full email is designed to be parsed.

Parsing the URL will return the bucket name and the path (referred to as the key by AWS). You can then use the bucket and key with the AWS SDK or library of your choice to take action with the file.

More information

As always there's information in the documentation and we're always looking for feedback. Feel free to Contact Us to join the beta and if you have any questions or comments!