Part 2: How to receive domain specific email on Gmail with Amazon SES

In an earlier post, I shared steps to send email with your own domain-specific email address using Gmail, through Amazon Simple Email Service (SES). Today, we will look at how to set up specific functions to receive domain-specific email on Gmail with Amazon SES. Do note that it’s more complicated than setting up the send email function. I relied largely on this article by Dani Loaz,  which was quite comprehensive. That said, I suggest that you reference The Travelling Squid’s article for the right code. It took me about two weeks to figure out the configuration, with the help of Amazon Support. The code used here was gathered from GitHub. For more information, you may wish to reference this GitHub page. There are several articles on troubleshooting as well.

How to receive domain specific email on Gmail with Amazon SES

*Note: The steps here assume that you have already set up the Amazon SES settings to send email via Gmail, based on the steps in this post.

1. Set up a Amazon S3 bucket

Set up a Amazon S3 bucket
Set up a Amazon S3 bucket – key in your bucket name in the field below and click next.

Go to the Amazon S3 homepage. Click on the tab ‘Create a bucket’ at the Amazon S3 page. Type in the name of your bucket. For step 2 (Configure options) , step 3 (Set permissions), step 4 (Review) I left the settings as default.

amazon s3 bucket
Under the Amazon S3 homepage, click on your bucket.
Amazon s3 bucket policy
Click on the Permissions tab, followed by Bucket Policy. Enter the code below and click save.

Under ‘insertyourbucketname’, type in the name of your S3 bucket. Under ‘insertyourAWSaccountID’ include your 12 digit AWS account ID. It can be found by clicking on the Account tab of your AWS account. Enter the code and click save.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GiveSESPermissionToWriteEmail",
            "Effect": "Allow",
            "Principal": {
                "Service": "ses.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::insertyourbucketname/*",
            "Condition": {
                "StringEquals": {
                    "aws:Referer": "insertyourAWSaccountID"
                }
            }
        }
    ]
}

2. Create a Lambda function, which will forward the emails from Amazon SES to your gmail account.

AWS lambda create function
Create a function with AWS Lambda – orange button on the right.
domain specific email on Gmail with Amazon SES AWS Lambda enter code
Enter the code below and click ‘Save’.

This step is somewhat tricky – it took me many hours to get the Lambda function up and running. First, go to Amazon Lambda. Click on ‘Create Function’. Enter the code below. Note: I have tweaked the code from Dani Loaz’s example, as the earlier execution was unsuccessful. Customise the code for these areas:

  • Insert your domain specific email address. (In my case, it was phebe@thetravellingsquid.com)
  • Insert the name of your S3 email bucket
  • Insert your gmail address
"use strict";

var AWS = require('aws-sdk');

console.log("AWS Lambda SES Forwarder // @arithmetric // Version 4.2.0");

// Configure the S3 bucket and key prefix for stored raw emails, and the
// mapping of email addresses to forward from and to.
//
// Expected keys/values:
//
// - fromEmail: Forwarded emails will come from this verified address
//
// - subjectPrefix: Forwarded emails subject will contain this prefix
//
// - emailBucket: S3 bucket name where SES stores emails.
//
// - emailKeyPrefix: S3 key name prefix where SES stores email. Include the
//   trailing slash.
//
// - forwardMapping: Object where the key is the lowercase email address from
//   which to forward and the value is an array of email addresses to which to
//   send the message.
//
//   To match all email addresses on a domain, use a key without the name part
//   of an email address before the "at" symbol (i.e. `@example.com`).
//
//   To match a mailbox name on all domains, use a key without the "at" symbol
//   and domain part of an email address (i.e. `info`).
var defaultConfig = {
  fromEmail: "insert your domain specific email address",
  subjectPrefix: "",
  emailBucket: "insert the name of your S3 email bucket",
 emailKeyPrefix: "",
  forwardMapping: {
 "insert your domain specific email address": [
 "insert your gmail address"
    ]
  }
};

/**
 * Parses the SES event record provided for the `mail` and `receipients` data.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.parseEvent = function(data) {
  // Validate characteristics of a SES event record.
  if (!data.event ||
      !data.event.hasOwnProperty('Records') ||
      data.event.Records.length !== 1 ||
      !data.event.Records[0].hasOwnProperty('eventSource') ||
      data.event.Records[0].eventSource !== 'aws:ses' ||
      data.event.Records[0].eventVersion !== '1.0') {
    data.log({message: "parseEvent() received invalid SES message:",
      level: "error", event: JSON.stringify(data.event)});
    return Promise.reject(new Error('Error: Received invalid SES message.'));
  }

  data.email = data.event.Records[0].ses.mail;
  data.recipients = data.event.Records[0].ses.receipt.recipients;
  return Promise.resolve(data);
};

/**
 * Transforms the original recipients to the desired forwarded destinations.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.transformRecipients = function(data) {
  var newRecipients = [];
  data.originalRecipients = data.recipients;
  data.recipients.forEach(function(origEmail) {
    var origEmailKey = origEmail.toLowerCase();
    if (data.config.forwardMapping.hasOwnProperty(origEmailKey)) {
      newRecipients = newRecipients.concat(
        data.config.forwardMapping[origEmailKey]);
      data.originalRecipient = origEmail;
    } else {
      var origEmailDomain;
      var origEmailUser;
      var pos = origEmailKey.lastIndexOf("@");
      if (pos === -1) {
        origEmailUser = origEmailKey;
      } else {
        origEmailDomain = origEmailKey.slice(pos);
        origEmailUser = origEmailKey.slice(0, pos);
      }
      if (origEmailDomain &&
          data.config.forwardMapping.hasOwnProperty(origEmailDomain)) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping[origEmailDomain]);
        data.originalRecipient = origEmail;
      } else if (origEmailUser &&
        data.config.forwardMapping.hasOwnProperty(origEmailUser)) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping[origEmailUser]);
        data.originalRecipient = origEmail;
      }
    }
  });

  if (!newRecipients.length) {
    data.log({message: "Finishing process. No new recipients found for " +
      "original destinations: " + data.originalRecipients.join(", "),
      level: "info"});
    return data.callback();
  }

  data.recipients = newRecipients;
  return Promise.resolve(data);
};

/**
 * Fetches the message data from S3.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.fetchMessage = function(data) {
  // Copying email object to ensure read permission
  data.log({level: "info", message: "Fetching email at s3://" +
    data.config.emailBucket + '/' + data.config.emailKeyPrefix +
    data.email.messageId});
  return new Promise(function(resolve, reject) {
    data.s3.copyObject({
      Bucket: data.config.emailBucket,
      CopySource: data.config.emailBucket + '/' + data.config.emailKeyPrefix +
        data.email.messageId,
      Key: data.config.emailKeyPrefix + data.email.messageId,
      ACL: 'private',
      ContentType: 'text/plain',
      StorageClass: 'STANDARD'
    }, function(err) {
      if (err) {
        data.log({level: "error", message: "copyObject() returned error:",
          error: err, stack: err.stack});
        return reject(
          new Error("Error: Could not make readable copy of email."));
      }

      // Load the raw email from S3
      data.s3.getObject({
        Bucket: data.config.emailBucket,
        Key: data.config.emailKeyPrefix + data.email.messageId
      }, function(err, result) {
        if (err) {
          data.log({level: "error", message: "getObject() returned error:",
            error: err, stack: err.stack});
          return reject(
            new Error("Error: Failed to load message body from S3."));
        }
        data.emailData = result.Body.toString();
        return resolve(data);
      });
    });
  });
};

/**
 * Processes the message data, making updates to recipients and other headers
 * before forwarding message.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.processMessage = function(data) {
  var match = data.emailData.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m);
  var header = match && match[1] ? match[1] : data.emailData;
  var body = match && match[2] ? match[2] : '';

  // Add "Reply-To:" with the "From" address if it doesn't already exists
  if (!/^Reply-To: /mi.test(header)) {
    match = header.match(/^From: (.*(?:\r?\n\s+.*)*\r?\n)/m);
    var from = match && match[1] ? match[1] : '';
    if (from) {
      header = header + 'Reply-To: ' + from;
      data.log({level: "info", message: "Added Reply-To address of: " + from});
    } else {
      data.log({level: "info", message: "Reply-To address not added because " +
       "From address was not properly extracted."});
    }
  }

  // SES does not allow sending messages from an unverified address,
  // so replace the message's "From:" header with the original
  // recipient (which is a verified domain)
  header = header.replace(
    /^From: (.*(?:\r?\n\s+.*)*)/mg,
    function(match, from) {
      var fromText;
      if (data.config.fromEmail) {
        fromText = 'From: ' + from.replace(/<(.*)>/, '').trim() +
        ' <' + data.config.fromEmail + '>';
      } else {
        fromText = 'From: ' + from.replace('<', 'at ').replace('>', '') +
        ' <' + data.originalRecipient + '>';
      }
      return fromText;
    });

  // Add a prefix to the Subject
  if (data.config.subjectPrefix) {
    header = header.replace(
      /^Subject: (.*)/mg,
      function(match, subject) {
        return 'Subject: ' + data.config.subjectPrefix + subject;
      });
  }

  // Replace original 'To' header with a manually defined one
  if (data.config.toEmail) {
    header = header.replace(/^To: (.*)/mg, () => 'To: ' + data.config.toEmail);
  }

  // Remove the Return-Path header.
  header = header.replace(/^Return-Path: (.*)\r?\n/mg, '');

  // Remove Sender header.
  header = header.replace(/^Sender: (.*)\r?\n/mg, '');

  // Remove Message-ID header.
  header = header.replace(/^Message-ID: (.*)\r?\n/mig, '');

  // Remove all DKIM-Signature headers to prevent triggering an
  // "InvalidParameterValue: Duplicate header 'DKIM-Signature'" error.
  // These signatures will likely be invalid anyways, since the From
  // header was modified.
  header = header.replace(/^DKIM-Signature: .*\r?\n(\s+.*\r?\n)*/mg, '');

  data.emailData = header + body;
  return Promise.resolve(data);
};

/**
 * Send email using the SES sendRawEmail command.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.sendMessage = function(data) {
  var params = {
    Destinations: data.recipients,
    Source: data.originalRecipient,
    RawMessage: {
      Data: data.emailData
    }
  };
  data.log({level: "info", message: "sendMessage: Sending email via SES. " +
    "Original recipients: " + data.originalRecipients.join(", ") +
    ". Transformed recipients: " + data.recipients.join(", ") + "."});
  return new Promise(function(resolve, reject) {
    data.ses.sendRawEmail(params, function(err, result) {
      if (err) {
        data.log({level: "error", message: "sendRawEmail() returned error.",
          error: err, stack: err.stack});
        return reject(new Error('Error: Email sending failed.'));
      }
      data.log({level: "info", message: "sendRawEmail() successful.",
        result: result});
      resolve(data);
    });
  });
};

/**
 * Handler function to be invoked by AWS Lambda with an inbound SES email as
 * the event.
 *
 * @param {object} event - Lambda event from inbound email received by AWS SES.
 * @param {object} context - Lambda context object.
 * @param {object} callback - Lambda callback object.
 * @param {object} overrides - Overrides for the default data, including the
 * configuration, SES object, and S3 object.
 */
exports.handler = function(event, context, callback, overrides) {
  var steps = overrides && overrides.steps ? overrides.steps :
  [
    exports.parseEvent,
    exports.transformRecipients,
    exports.fetchMessage,
    exports.processMessage,
    exports.sendMessage
  ];
  var data = {
    event: event,
    callback: callback,
    context: context,
    config: overrides && overrides.config ? overrides.config : defaultConfig,
    log: overrides && overrides.log ? overrides.log : console.log,
    ses: overrides && overrides.ses ? overrides.ses : new AWS.SES(),
    s3: overrides && overrides.s3 ?
      overrides.s3 : new AWS.S3({signatureVersion: 'v4'})
  };
  Promise.series(steps, data)
    .then(function(data) {
      data.log({level: "info", message: "Process finished successfully."});
      return data.callback();
    })
    .catch(function(err) {
      data.log({level: "error", message: "Step returned error: " + err.message,
        error: err, stack: err.stack});
      return data.callback(new Error("Error: Step returned error."));
    });
};

Promise.series = function(promises, initValue) {
  return promises.reduce(function(chain, promise) {
    if (typeof promise !== 'function') {
      return Promise.reject(new Error("Error: Invalid promise item: " +
        promise));
    }
    return chain.then(promise);
  }, Promise.resolve(initValue));
};

When you are done, click on the ‘Save’ button on the top right hand side of the page.

3. Configure a test event for your Lambda function

AWS Lambda configure test event
Next, configure a test event for your function. A test event is required for your function to be executed successfully. The default ‘Hello World’ test event will not work for this function. Click on ‘Configure test event’ at the top right hand corner. Your Lambda ARN address is directly on top (grey box). Take note of that as you will need to add it to the test event.
domain specific email on Gmail with Amazon SES AWS lambda test event with code
Enter the code below into the AWS Lambda test event.

Configure the test event with this code. It came directly from Github. The only change I made was to add the Amazon Resource Names (ARN) address of my Lambda function under ‘functionARN’. Once you’re done, click create.

 {
  "Records": [
    {
      "eventVersion": "1.0",
      "ses": {
        "mail": {
          "commonHeaders": {
            "from": [
              "Jane Doe <janedoe@example.com>"
            ],
            "to": [
              "info@example.com"
            ],
            "returnPath": "janedoe@example.com",
            "messageId": "<0123456789example.com>",
            "date": "Wed, 7 Oct 2015 12:34:56 -0700",
            "subject": "Test Subject"
          },
          "source": "janedoe@example.com",
          "timestamp": "1970-01-01T00:00:00.000Z",
          "destination": [
            "info@example.com"
          ],
          "headers": [
            {
              "name": "Return-Path",
              "value": "<janedoe@example.com>"
            },
            {
              "name": "Received",
              "value": "from mailer.example.com (mailer.example.com [203.0.113.1]) by inbound-smtp.us-west-2.amazonaws.com with SMTP id o3vrnil0e2ic28trm7dfk3kc2v0cnbeccl4nbp0g1 for johndoe@example.com; Wed, 07 Oct 2015 12:34:56 +0000 (UTC)"
            },
            {
              "name": "DKIM-Signature",
              "value": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=example.com; s=example; h=mime-version:from:date:message-id:subject:to:content-type; bh=jX3F0bCAI7sIbkHyy3mLYO28ieImz2R0P8HwQkklFj4=; b=sQwJ+LMe9RjkesGu+vqU56asvMhkwoRYrWCbVt6WJulueecwfEwRf9JVWgkBTKiL6m2hr70xDbPWDhtLdLO+jB3hzjVnXwK3pYIOHw3vxG6NtJ6o61XSUwjEsp9tdyxQjZf2HNYee873832l3K1EeSXKzxYk9Pwqcpi3dMC74ct9GukjIevf1H4l3b1L2d9VYTL0LGZGHOAyMnHmEGB8ZExWbI+k6khpurTQQ4sp4PZPRlgHtnj3Zzv7nmpTo7dtPG5z5S9J+L+Ba7dixT0jn3HuhaJ9b+VThboo4YfsX9PMNhWWxGjVksSFOcGluPO7QutCPyoY4gbxtwkN9W69HA=="
            },
            {
              "name": "MIME-Version",
              "value": "1.0"
            },
            {
              "name": "From",
              "value": "Jane Doe <janedoe@example.com>"
            },
            {
              "name": "Date",
              "value": "Wed, 7 Oct 2015 12:34:56 -0700"
            },
            {
              "name": "Message-ID",
              "value": "<0123456789example.com>"
            },
            {
              "name": "Subject",
              "value": "Test Subject"
            },
            {
              "name": "To",
              "value": "info@example.com"
            },
            {
              "name": "Content-Type",
              "value": "text/plain; charset=UTF-8"
            }
          ],
          "headersTruncated": false,
          "messageId": "o3vrnil0e2ic28trm7dakrc2v0clambda4nbp0g1"
        },
        "receipt": {
          "recipients": [
            "info@example.com"
          ],
          "timestamp": "1970-01-01T00:00:00.000Z",
          "spamVerdict": {
            "status": "PASS"
          },
          "dkimVerdict": {
            "status": "PASS"
          },
          "processingTimeMillis": 574,
          "action": {
            "type": "Lambda",
            "invocationType": "Event",
            "functionArn": "arn:aws:lambda:INCLUDE THE ARN FOR YOUR LAMBDA FUNCTION HERE"
          },
          "spfVerdict": {
            "status": "PASS"
          },
          "virusVerdict": {
            "status": "PASS"
          }
        }
      },
      "eventSource": "aws:ses"
    }
  ]
}

4. Include the necessary IAM permission for your Lambda Function

domain specific email on Gmail with Amazon SES
On the same page of your Lambda function, scroll down to view ‘Execution Role’. Click on the link to view the ‘test role’ on the IAM Console.

For your Lambda function to execute successfully, you will need to provide it with the necessary permissions through AWS Identity and Access Management (IAM). Under the Lambda function page, scroll down and you will see the ‘Execution Role’ box near the bottom of the page. Click on the link which says, ‘View the <name of lambda function> role on IAM’ (box in red).

It will bring you to the page below. Click on the policy name – it should read ‘AWSLambdaBasicExecutionRole…’ Click on Edit Policy and the JSON (Java Script Object Notification) tab.

AWS Lambda IAM role
Click on the ‘Execution Role’ (Link in red box)
AWS Lambda IAM policy
Under Permissions, click Edit Policy.

Click on the ‘JSON’ tab and enter the following code. The only edit I made was to the arn address of my Amazon S3 bucket, under Resource.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": "ses:SendRawEmail",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::INSERT S3 BUCKET NAME HERE/*"
        }
    ]
}

5. Create a rule set under Amazon SES

Under the Email Receiving category, create a rule set. Verify the domain-specific email address you will like to receive emails from.

Rule set verification status.
Verify the domain-specific email which you would like to receive email from. In my case, it was phebe@thetravellingsquid.com

You will reach the Action section.

domain specific email on Gmail with Amazon SES Amazon ses s3 lambda rule sets
Create a rule set for Amazon s3 and Lambda. Connect them to the S3 bucket and Lambda function you had set up in Steps 1 and 2. The settings should look something like that.

6. Test out your Lambda function(and hope for the best)

domain specific email on Gmail with Amazon SES AWS lambda function successful execution
Execution was finally successful!

I failed this step many times and received the message ‘Step Returned Error’. I realised that this was because the test event in Step 3 had not been configured properly. I also made some further tweaks to the code by Dani Loaz. The execution was successful but there were some error logs. I managed to get my emails forwarded to my Gmail account, so that didn’t matter.

If the Lambda test function works, send a test email to your domain-specific email account and see if it gets forwarded to your Gmail account. You might need to troubleshoot specific error codes further. I recommend checking the Github site for solutions. Alternatively, if all else fails, you could enlist AWS Support for USD29 a month. They  were very prompt during the troubleshooting process and I learnt alot.

The Travelling Squid’s Take

receive domain specific email on Gmail with Amazon SES
An instastory depicting my frustration with the error codes!

I found the process to receive domain specific email on Gmail with Amazon SES most challenging and it took me 2 weeks to get it going. I also learnt that small errors in a line of code could result in an execution error, so you got to be quite detailed.

If the Lambda function does not execute correctly, an interim measure is to use Mozilla Thunderbird to view your emails from your S3 bucket. That involves downloading individual objects and adding the .eml extension at the back of the file. The downside of this approach is that you’ve got to download individual objects. Many of them are spam too.

Hope you found the steps to receive domain specific email on Gmail with Amazon SES useful. If you have questions, feel free to drop a comment in the box below.

[Note: We have been alerted that the email address phebe@thetravellingsquid.com has been spoofed, i.e., a person or program has been masquerading us by sending spam emails tagged to our email address. We wish to clarify that the emails have not been sent by us. Please ignore suspicious emails especially those on erectile dysfunction, bitcoins, etc and DO NOT click on any of the links in the email. Delete them immediately. Thank you.]

Leave a Reply

Your email address will not be published. Required fields are marked *

Read previous post:
Wordpress site on Lightsail goes down due to insufficient disk space
How to plan a vacation in the US

In some fairytale world, there are people who are on vacation all the time, traveling to different sites and taking...

Close