Part 2: How to receive domain specific email on Gmail with Amazon SES
In an earlier post, I shared steps to send email with your own domain-specific email address using Gmail, through Amazon Simple Email Service (SES). Today, we will look at how to set up specific functions to receive domain-specific email on Gmail with Amazon SES. Do note that it’s more complicated than setting up the send email function. I relied largely on this article by Dani Loaz, which was quite comprehensive. That said, I suggest that you reference The Travelling Squid’s article for the right code. It took me about two weeks to figure out the configuration, with the help of Amazon Support. The code used here was gathered from GitHub. For more information, you may wish to reference this GitHub page. There are several articles on troubleshooting as well.
How to receive domain specific email on Gmail with Amazon SES
*Note: The steps here assume that you have already set up the Amazon SES settings to send email via Gmail, based on the steps in this post.
1. Set up a Amazon S3 bucket
Go to the Amazon S3 homepage. Click on the tab ‘Create a bucket’ at the Amazon S3 page. Type in the name of your bucket. For step 2 (Configure options) , step 3 (Set permissions), step 4 (Review) I left the settings as default.
Under ‘insertyourbucketname’, type in the name of your S3 bucket. Under ‘insertyourAWSaccountID’ include your 12 digit AWS account ID. It can be found by clicking on the Account tab of your AWS account. Enter the code and click save.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GiveSESPermissionToWriteEmail",
"Effect": "Allow",
"Principal": {
"Service": "ses.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::insertyourbucketname/*",
"Condition": {
"StringEquals": {
"aws:Referer": "insertyourAWSaccountID"
}
}
}
]
}
2. Create a Lambda function, which will forward the emails from Amazon SES to your gmail account.
This step is somewhat tricky – it took me many hours to get the Lambda function up and running. First, go to Amazon Lambda. Click on ‘Create Function’. Enter the code below. Note: I have tweaked the code from Dani Loaz’s example, as the earlier execution was unsuccessful. Customise the code for these areas:
- Insert your domain specific email address. (In my case, it was phebe@thetravellingsquid.com)
- Insert the name of your S3 email bucket
- Insert your gmail address
"use strict";
var AWS = require('aws-sdk');
console.log("AWS Lambda SES Forwarder // @arithmetric // Version 4.2.0");
// Configure the S3 bucket and key prefix for stored raw emails, and the
// mapping of email addresses to forward from and to.
//
// Expected keys/values:
//
// - fromEmail: Forwarded emails will come from this verified address
//
// - subjectPrefix: Forwarded emails subject will contain this prefix
//
// - emailBucket: S3 bucket name where SES stores emails.
//
// - emailKeyPrefix: S3 key name prefix where SES stores email. Include the
// trailing slash.
//
// - forwardMapping: Object where the key is the lowercase email address from
// which to forward and the value is an array of email addresses to which to
// send the message.
//
// To match all email addresses on a domain, use a key without the name part
// of an email address before the "at" symbol (i.e. `@example.com`).
//
// To match a mailbox name on all domains, use a key without the "at" symbol
// and domain part of an email address (i.e. `info`).
var defaultConfig = {
fromEmail: "insert your domain specific email address",
subjectPrefix: "",
emailBucket: "insert the name of your S3 email bucket",
emailKeyPrefix: "",
forwardMapping: {
"insert your domain specific email address": [
"insert your gmail address"
]
}
};
/**
* Parses the SES event record provided for the `mail` and `receipients` data.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.parseEvent = function(data) {
// Validate characteristics of a SES event record.
if (!data.event ||
!data.event.hasOwnProperty('Records') ||
data.event.Records.length !== 1 ||
!data.event.Records[0].hasOwnProperty('eventSource') ||
data.event.Records[0].eventSource !== 'aws:ses' ||
data.event.Records[0].eventVersion !== '1.0') {
data.log({message: "parseEvent() received invalid SES message:",
level: "error", event: JSON.stringify(data.event)});
return Promise.reject(new Error('Error: Received invalid SES message.'));
}
data.email = data.event.Records[0].ses.mail;
data.recipients = data.event.Records[0].ses.receipt.recipients;
return Promise.resolve(data);
};
/**
* Transforms the original recipients to the desired forwarded destinations.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.transformRecipients = function(data) {
var newRecipients = [];
data.originalRecipients = data.recipients;
data.recipients.forEach(function(origEmail) {
var origEmailKey = origEmail.toLowerCase();
if (data.config.forwardMapping.hasOwnProperty(origEmailKey)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailKey]);
data.originalRecipient = origEmail;
} else {
var origEmailDomain;
var origEmailUser;
var pos = origEmailKey.lastIndexOf("@");
if (pos === -1) {
origEmailUser = origEmailKey;
} else {
origEmailDomain = origEmailKey.slice(pos);
origEmailUser = origEmailKey.slice(0, pos);
}
if (origEmailDomain &&
data.config.forwardMapping.hasOwnProperty(origEmailDomain)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailDomain]);
data.originalRecipient = origEmail;
} else if (origEmailUser &&
data.config.forwardMapping.hasOwnProperty(origEmailUser)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailUser]);
data.originalRecipient = origEmail;
}
}
});
if (!newRecipients.length) {
data.log({message: "Finishing process. No new recipients found for " +
"original destinations: " + data.originalRecipients.join(", "),
level: "info"});
return data.callback();
}
data.recipients = newRecipients;
return Promise.resolve(data);
};
/**
* Fetches the message data from S3.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.fetchMessage = function(data) {
// Copying email object to ensure read permission
data.log({level: "info", message: "Fetching email at s3://" +
data.config.emailBucket + '/' + data.config.emailKeyPrefix +
data.email.messageId});
return new Promise(function(resolve, reject) {
data.s3.copyObject({
Bucket: data.config.emailBucket,
CopySource: data.config.emailBucket + '/' + data.config.emailKeyPrefix +
data.email.messageId,
Key: data.config.emailKeyPrefix + data.email.messageId,
ACL: 'private',
ContentType: 'text/plain',
StorageClass: 'STANDARD'
}, function(err) {
if (err) {
data.log({level: "error", message: "copyObject() returned error:",
error: err, stack: err.stack});
return reject(
new Error("Error: Could not make readable copy of email."));
}
// Load the raw email from S3
data.s3.getObject({
Bucket: data.config.emailBucket,
Key: data.config.emailKeyPrefix + data.email.messageId
}, function(err, result) {
if (err) {
data.log({level: "error", message: "getObject() returned error:",
error: err, stack: err.stack});
return reject(
new Error("Error: Failed to load message body from S3."));
}
data.emailData = result.Body.toString();
return resolve(data);
});
});
});
};
/**
* Processes the message data, making updates to recipients and other headers
* before forwarding message.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.processMessage = function(data) {
var match = data.emailData.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m);
var header = match && match[1] ? match[1] : data.emailData;
var body = match && match[2] ? match[2] : '';
// Add "Reply-To:" with the "From" address if it doesn't already exists
if (!/^Reply-To: /mi.test(header)) {
match = header.match(/^From: (.*(?:\r?\n\s+.*)*\r?\n)/m);
var from = match && match[1] ? match[1] : '';
if (from) {
header = header + 'Reply-To: ' + from;
data.log({level: "info", message: "Added Reply-To address of: " + from});
} else {
data.log({level: "info", message: "Reply-To address not added because " +
"From address was not properly extracted."});
}
}
// SES does not allow sending messages from an unverified address,
// so replace the message's "From:" header with the original
// recipient (which is a verified domain)
header = header.replace(
/^From: (.*(?:\r?\n\s+.*)*)/mg,
function(match, from) {
var fromText;
if (data.config.fromEmail) {
fromText = 'From: ' + from.replace(/<(.*)>/, '').trim() +
' <' + data.config.fromEmail + '>';
} else {
fromText = 'From: ' + from.replace('<', 'at ').replace('>', '') +
' <' + data.originalRecipient + '>';
}
return fromText;
});
// Add a prefix to the Subject
if (data.config.subjectPrefix) {
header = header.replace(
/^Subject: (.*)/mg,
function(match, subject) {
return 'Subject: ' + data.config.subjectPrefix + subject;
});
}
// Replace original 'To' header with a manually defined one
if (data.config.toEmail) {
header = header.replace(/^To: (.*)/mg, () => 'To: ' + data.config.toEmail);
}
// Remove the Return-Path header.
header = header.replace(/^Return-Path: (.*)\r?\n/mg, '');
// Remove Sender header.
header = header.replace(/^Sender: (.*)\r?\n/mg, '');
// Remove Message-ID header.
header = header.replace(/^Message-ID: (.*)\r?\n/mig, '');
// Remove all DKIM-Signature headers to prevent triggering an
// "InvalidParameterValue: Duplicate header 'DKIM-Signature'" error.
// These signatures will likely be invalid anyways, since the From
// header was modified.
header = header.replace(/^DKIM-Signature: .*\r?\n(\s+.*\r?\n)*/mg, '');
data.emailData = header + body;
return Promise.resolve(data);
};
/**
* Send email using the SES sendRawEmail command.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.sendMessage = function(data) {
var params = {
Destinations: data.recipients,
Source: data.originalRecipient,
RawMessage: {
Data: data.emailData
}
};
data.log({level: "info", message: "sendMessage: Sending email via SES. " +
"Original recipients: " + data.originalRecipients.join(", ") +
". Transformed recipients: " + data.recipients.join(", ") + "."});
return new Promise(function(resolve, reject) {
data.ses.sendRawEmail(params, function(err, result) {
if (err) {
data.log({level: "error", message: "sendRawEmail() returned error.",
error: err, stack: err.stack});
return reject(new Error('Error: Email sending failed.'));
}
data.log({level: "info", message: "sendRawEmail() successful.",
result: result});
resolve(data);
});
});
};
/**
* Handler function to be invoked by AWS Lambda with an inbound SES email as
* the event.
*
* @param {object} event - Lambda event from inbound email received by AWS SES.
* @param {object} context - Lambda context object.
* @param {object} callback - Lambda callback object.
* @param {object} overrides - Overrides for the default data, including the
* configuration, SES object, and S3 object.
*/
exports.handler = function(event, context, callback, overrides) {
var steps = overrides && overrides.steps ? overrides.steps :
[
exports.parseEvent,
exports.transformRecipients,
exports.fetchMessage,
exports.processMessage,
exports.sendMessage
];
var data = {
event: event,
callback: callback,
context: context,
config: overrides && overrides.config ? overrides.config : defaultConfig,
log: overrides && overrides.log ? overrides.log : console.log,
ses: overrides && overrides.ses ? overrides.ses : new AWS.SES(),
s3: overrides && overrides.s3 ?
overrides.s3 : new AWS.S3({signatureVersion: 'v4'})
};
Promise.series(steps, data)
.then(function(data) {
data.log({level: "info", message: "Process finished successfully."});
return data.callback();
})
.catch(function(err) {
data.log({level: "error", message: "Step returned error: " + err.message,
error: err, stack: err.stack});
return data.callback(new Error("Error: Step returned error."));
});
};
Promise.series = function(promises, initValue) {
return promises.reduce(function(chain, promise) {
if (typeof promise !== 'function') {
return Promise.reject(new Error("Error: Invalid promise item: " +
promise));
}
return chain.then(promise);
}, Promise.resolve(initValue));
};
When you are done, click on the ‘Save’ button on the top right hand side of the page.
3. Configure a test event for your Lambda function
Configure the test event with this code. It came directly from Github. The only change I made was to add the Amazon Resource Names (ARN) address of my Lambda function under ‘functionARN’. Once you’re done, click create.
{
"Records": [
{
"eventVersion": "1.0",
"ses": {
"mail": {
"commonHeaders": {
"from": [
"Jane Doe <janedoe@example.com>"
],
"to": [
"info@example.com"
],
"returnPath": "janedoe@example.com",
"messageId": "<0123456789example.com>",
"date": "Wed, 7 Oct 2015 12:34:56 -0700",
"subject": "Test Subject"
},
"source": "janedoe@example.com",
"timestamp": "1970-01-01T00:00:00.000Z",
"destination": [
"info@example.com"
],
"headers": [
{
"name": "Return-Path",
"value": "<janedoe@example.com>"
},
{
"name": "Received",
"value": "from mailer.example.com (mailer.example.com [203.0.113.1]) by inbound-smtp.us-west-2.amazonaws.com with SMTP id o3vrnil0e2ic28trm7dfk3kc2v0cnbeccl4nbp0g1 for johndoe@example.com; Wed, 07 Oct 2015 12:34:56 +0000 (UTC)"
},
{
"name": "DKIM-Signature",
"value": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=example.com; s=example; h=mime-version:from:date:message-id:subject:to:content-type; bh=jX3F0bCAI7sIbkHyy3mLYO28ieImz2R0P8HwQkklFj4=; b=sQwJ+LMe9RjkesGu+vqU56asvMhkwoRYrWCbVt6WJulueecwfEwRf9JVWgkBTKiL6m2hr70xDbPWDhtLdLO+jB3hzjVnXwK3pYIOHw3vxG6NtJ6o61XSUwjEsp9tdyxQjZf2HNYee873832l3K1EeSXKzxYk9Pwqcpi3dMC74ct9GukjIevf1H4l3b1L2d9VYTL0LGZGHOAyMnHmEGB8ZExWbI+k6khpurTQQ4sp4PZPRlgHtnj3Zzv7nmpTo7dtPG5z5S9J+L+Ba7dixT0jn3HuhaJ9b+VThboo4YfsX9PMNhWWxGjVksSFOcGluPO7QutCPyoY4gbxtwkN9W69HA=="
},
{
"name": "MIME-Version",
"value": "1.0"
},
{
"name": "From",
"value": "Jane Doe <janedoe@example.com>"
},
{
"name": "Date",
"value": "Wed, 7 Oct 2015 12:34:56 -0700"
},
{
"name": "Message-ID",
"value": "<0123456789example.com>"
},
{
"name": "Subject",
"value": "Test Subject"
},
{
"name": "To",
"value": "info@example.com"
},
{
"name": "Content-Type",
"value": "text/plain; charset=UTF-8"
}
],
"headersTruncated": false,
"messageId": "o3vrnil0e2ic28trm7dakrc2v0clambda4nbp0g1"
},
"receipt": {
"recipients": [
"info@example.com"
],
"timestamp": "1970-01-01T00:00:00.000Z",
"spamVerdict": {
"status": "PASS"
},
"dkimVerdict": {
"status": "PASS"
},
"processingTimeMillis": 574,
"action": {
"type": "Lambda",
"invocationType": "Event",
"functionArn": "arn:aws:lambda:INCLUDE THE ARN FOR YOUR LAMBDA FUNCTION HERE"
},
"spfVerdict": {
"status": "PASS"
},
"virusVerdict": {
"status": "PASS"
}
}
},
"eventSource": "aws:ses"
}
]
}
4. Include the necessary IAM permission for your Lambda Function
For your Lambda function to execute successfully, you will need to provide it with the necessary permissions through AWS Identity and Access Management (IAM). Under the Lambda function page, scroll down and you will see the ‘Execution Role’ box near the bottom of the page. Click on the link which says, ‘View the <name of lambda function> role on IAM’ (box in red).
It will bring you to the page below. Click on the policy name – it should read ‘AWSLambdaBasicExecutionRole…’ Click on Edit Policy and the JSON (Java Script Object Notification) tab.
Click on the ‘JSON’ tab and enter the following code. The only edit I made was to the arn address of my Amazon S3 bucket, under Resource.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": "ses:SendRawEmail",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::INSERT S3 BUCKET NAME HERE/*"
}
]
}
5. Create a rule set under Amazon SES
Under the Email Receiving category, create a rule set. Verify the domain-specific email address you will like to receive emails from.
You will reach the Action section.
6. Test out your Lambda function(and hope for the best)
I failed this step many times and received the message ‘Step Returned Error’. I realised that this was because the test event in Step 3 had not been configured properly. I also made some further tweaks to the code by Dani Loaz. The execution was successful but there were some error logs. I managed to get my emails forwarded to my Gmail account, so that didn’t matter.
If the Lambda test function works, send a test email to your domain-specific email account and see if it gets forwarded to your Gmail account. You might need to troubleshoot specific error codes further. I recommend checking the Github site for solutions. Alternatively, if all else fails, you could enlist AWS Support for USD29 a month. They were very prompt during the troubleshooting process and I learnt alot.
The Travelling Squid’s Take
I found the process to receive domain specific email on Gmail with Amazon SES most challenging and it took me 2 weeks to get it going. I also learnt that small errors in a line of code could result in an execution error, so you got to be quite detailed.
If the Lambda function does not execute correctly, an interim measure is to use Mozilla Thunderbird to view your emails from your S3 bucket. That involves downloading individual objects and adding the .eml extension at the back of the file. The downside of this approach is that you’ve got to download individual objects. Many of them are spam too.
Hope you found the steps to receive domain specific email on Gmail with Amazon SES useful. If you have questions, feel free to drop a comment in the box below.
[Note: We have been alerted that the email address phebe@thetravellingsquid.com has been spoofed, i.e., a person or program has been masquerading us by sending spam emails tagged to our email address. We wish to clarify that the emails have not been sent by us. Please ignore suspicious emails especially those on erectile dysfunction, bitcoins, etc and DO NOT click on any of the links in the email. Delete them immediately. Thank you.]
