New inbound feature - retries

Since launching Postmark Inbound, one of the features that our customers requested the most is the ability to retry inbound posts. Lot of things can happen that can disrupt an inbound JSON post coming from Postmark to the inbound hook server. The web application receiving emails or the web server hosting it can go down due to various reasons. Or, networking issues can render your inbound hook unreachable from Postmark.

Whatever the reason for the failed POST, you shouldn’t ever miss an email. That’s why we are adding automatic retries to Inbound, to ensure your web app receives the processed inbound emails.

How does it work?

When an email is sent to your inbound mailbox, Postmark will retrieve it and process it. Immediately, it will try to post the generated JSON to the inbound hook URL you defined. If the post was successful, it will be listed as Processed. The post is considered successful when the inbound hook returns one of the HTTP 200 codes. This is the same behavior we had until now and the presentation in the UI is the same.

However, if the hook returns a non-200 code, we will now schedule the JSON post for a retry. The entry in the activity screen will show how many retries were attempted, and will show the timestamp of the next pending retry. 

You can even filter your emails by using the drop-down filter, looking for “Retrying” emails.

A total of 10 retries will be made, with growing intervals from 1 minute to 6 hours. This is done so that if your web app has longer downtime, it still has a chance to receive all of the posts when it is back online. Only if all of the retries have failed, your Inbound page will show the message as Inbound Error. 

How much does retry cost?

Retries don’t cost you anything - a single credit is taken for processing inbound email and includes all of the potential retries.

What is the next thing we are working on?

We are improving our infrastructure to produce much faster processing times for inbound emails. Another idea we’re working on is manual retries initiated from UI. If you have any suggestions on what you would like to see with Inbound, please let us know! You can comment here, send us your thoughts over Twitter (@postmarkapp) or email support.

We have introduced some changes to the posted JSON object.

In a nutshell, here are the improvements:

• All contacts (From, To, Cc addresses) will contain both email and full name (display name). Before this change, we only extracted email address, but having the the full name will be a useful addition. Email and full name are separated into two JSON fields, so no additional parsing on your side is required.
   
• Emails with multiple recipients will be parsed on our side and passed with the JSON object as JSON arrays. Now you can easily traverse through all recipients, without any parsing.

• The Date property is added to JSON, holding the timestamp of the original message (presenting local time of the message sender). This can be useful to fetch local time zone of the people sending messages to your web app, among other things.

The specifics of these changes are included in the developer docs for handling contacts and send date.

Executing a non-breaking release

To be respectful of your apps, we are going to keep all existing fields in posted JSON and only add new ones. 

We added new fields with similar information to some of the existing fields, but they offer more information and have additional formatting. 

For example:

"From": "myUser@theirDomain.com",
"FromFull": {
  "Email": "myUser@theirDomain.com",
  "Name": "John Doe"
},

This will cause some duplicate data to appear in posted JSON but will allow you some time to change over to using the new format. 

We’re looking for specific use cases where the legacy format is useful. If your app includes one of those cases, please reply to this email and let us know. 

Finally, many thanks to the developers who have quickly updated their code examples for Postmark Inbound in response to this update. If you’re interested in providing examples in other languages/frameworks, we can offer free credits in return for your contributions!

We’re really happy to be hearing from customers about how they’ve been using Inbound, so if you’ve got stories you haven’t told yet, please share them with us!

Our friends at Server Density are running a series of deals in the coming months highlighting the useful sysadmin/development tools that they use and love. We’re thrilled to be their first featured deal.

In return, they’re offering a year of free server monitoring for 1 server to all Postmark customers!

What it does: Server Density provides hosted monitoring, analytics, and alerts for your servers and websites. You can easily start monitoring the status and response times of your websites which can be correlated with server metrics like CPU, memory, disk and other applications like Apache, MySQL and MongoDB.

How we use it: We use Server Density to monitor server health and statistics (on OS and DB levels). And we also publish custom metrics to Server Density which are used to monitor our own parameters in Postmark.

The deal: Server Density is offering Postmark customers 1 year of free server monitoring for 1 server to new accounts.

How to get the deal: E-mail hello@serverdensity.com with your Server Density account URL and mention this blog post.

I owe an explanation for the recent issues on Postmark. This week we had some pretty terrible performance on our MongoDB cluster, resulting in delayed emails, disabled activity and even disabled bounce API. No emails were lost during the outages, but I do want to explain what happened, what we did, and what we are doing about it.

Cause and effect

In short, one of our MongoDB clusters failed, forcing a failover to a secondary machine as a primary. Unfortunately this new primary did not have enough resources (Disk IO) to keep up with the write load that Postmark throws at it. Since the other replica failed, we basically had to wait until it rebuilt to bring it back, which took over a day with the amount of data we had on that cluster. Once we failed over again, everything instantly came back to normal.

The issue that occurred is not something new. We’ve had challenges with MongoDB in the past. I would say that it is not entirely about how MongoDB works, but how it fits our needs and how we use it. Postmark captures a lot of data each day, which we store both in MongoDB for archival and Elastic Search for searching.

We go through so much data that in order to maintain the performance that our customers expect, we need to purge data constantly. With a solution like MongoDB, this causes two problems.

First, we have to constantly delete data, which adds to the disk utilization on each primary in the replicaset. Second, when data is purged, it does not release the space on disk, meaning that we have to run constant compactions or resync to get it back.

As a result of these problems, our Bounce API was down for almost two days. We chose to disable that part of the API because it uses MongoDB queries to find emails, which puts additional load on the servers. We decided that it was better to have email sending services in good shape if it meant the Bounce API was offline.

While embarrassing, I still feel this was the right decision. The other side affect of the performance issues are inaccurate “In queue” messages in activity. While all emails were sent successfully (although possibly delayed in some cases), the activity statuses were still being pulled from MongoDB data which reflected some inconsistency while it was suffering. We are slowly updating the activity history to reflect the properly sent messages and you can expect them to be back to normal within the next day or so. All new messages are being shown accurately.

A silver lining

One really good thing that came out of the performance issues is an updated Bounce API. The bounce API has always gone directly to MongoDB when searching for email records. Since MongoDB is not really meant for searching, it was very slow, and if you use it you probably had timeouts in the past. One of our developers, JP Toto, was able to rebuild it to make calls directly to Elastic Search, which results in near instant responses. This change will also allow us to enhance how you search for bounce (and sent) records in the future.

Rethinking our architecture

We’ve been on MongoDB for a long time. At this point we are rethinking our architecture and have a plan for some very thorough benchmarking of other nosql (and sql) databases to see how they might fit our needs. The most important part of the results of this benchmarking is making sure that our email archive cannot affect the real-time sending that our customers rely on. This either means an eventually consistent solution or heavy background processing, or a combination of both. We still have not ruled out MongoDB, but we have some ideas on how we can use it differently that might fit us better.

Since most of our customers are also developers, we’ll make sure to post our benchmarking results once we are done.

More than anything, I want to sincerely apologize for the problems. For the most part, we know we are doing our job when you forget we exist because everything “just works”. We heard from many of you because that wasn‘t the case in the past few days.

We‘ll make sure that the next time we hear from you, it’s on a positive note. Thanks for your patience and support.

Today we were going to have a really important, but low impact maintenance. This morning at 9am the team brought the application offline to migrate to a new, more powerful database. Email sending was not going to be affected during most of the process, we expected about 2-4 minutes of paused sending. Things did not happen as we expected and we had some long outages and delays in sending. I want to explain what happened in detail.

You know that sending your transactional email through Postmark is the best way to improve your delivery rates. But sometimes, even with the best of tools and intentions, your emails will end up in spam and you won’t know why.

The next time you’re working to troubleshoot why an email is going to your (or your customers’) spam filter in Gmail, you can click on the email itself and look right below the sender and recipient information for more information about why Google thinks that email is spam.

These are just a couple of examples from my own spambox. Other options may include:

We’ve found that lots of messages from xyz@xyz.com are spam.

and

It contains content that’s typically used in spam messages

Even when you’re sending through Postmark, it’s a great idea to make sure that you are set up to periodically receive copies of your transactional emails, and take special notice  when you make copy changes. If emails start going to spam, you can use this new Gmail feature to try to determine why.

Of course, if you’re ever stumped as to why an email sent with Postmark isn’t being delivered or is landing in the spam filter, Postmark’s expert support team is ready to help you diagnose the problem. 

We encourage our customers to think of Postmark is a “good senders club” for transactional email. ISPs base part of a sender’s reputation on things like spam complaints and bounces and Postmark helps you make sure that you can keep those rates low in order to keep deliverability high for everyone.

A common technique in email marketing services is to help customers with “list hygiene”, largely for the same reasons. An ESP will often provide tools to help their customers scan the list for emails that are likely to generate bounces. This means that the tool will highlight emails with commonly fake or likely misspelled domains.

Transactional email requires a different approach, because a user signing up for a service is much less likely to be using an intentionally fake email address. They are, however, likely to have typos.

A common technique to avoid email typos is to have the customer type their email in twice and verify, but that’s extra work that can slow a user down during signup. Instead, we recommend doing a little bit of extra work to help your customers sign up quicker and still proactively catch email typos. 

Here are couple of techniques for catching and fixing typo’d emails for your users in a way that’s likely to surprise and delight them.

Client-Side Suggestion

A brand new jQuery library authored by the developers at Kicksend allows you to specify a set of domains that are often mistyped, and it will catch domains that are similar enough to be a typo and suggest the correct domain to the user.

The library does require that you bring your own list of domains, which you can get ideas for from browsing your bounces in the Postmark activity filter. 

This library - dubbed mailcheck - is new, so we strongly recommend testing it well before implementing it. Grab it on Github here.

Postmark Bounce Hooks

Have you ever had a user who didn’t get their confirmation email, only for you to realize after investigation that it was because they had a typo in their email address?  Or what if someone changes their email address and forgets to update it in your app, and then stops getting important notifications?

If you use Postmark for your emails, we’ll detects bounces and their cause, and provide information back to your app when the bounce is caught via a webhook, where you can parse information about the bounce from a JSON post. This information can be used any way you like, but one of the ways we recommend using it is to provide useful information to your logged in users.

As an example in Beanstalk, we parse bounces and notify users that are logged in about the bounce. We even make recommendations based on the bounce reason and recommend updating the address in Beanstalk while they are still logged in. 

If the user logged in is an account owner, we let them know about any addresses in their account that bounced recently.

With just a little bit of work we’re able to reduce customer support by a lot, and help keep our customers happy and informed.

Take a look at our bounce hooks API to set this up. 

Less bounces = happier customers

Remember that reducing your bounce rate isn’t only a best practice for being a Postmark customer, but your customers will appreciate you being thoughtful and helping them catch their mistakes. Reducing their confusion and frustration will result in them spending more time enjoying your application, and little experiences like these help a lot!

Hot on the heels of the Node.js example we shared last week, we recently had two new “mitt” libraries contributed to make it easy to process JSON data sent in Postmark Inbound’s webhooks.

NEW PHP Mitt:
Joffrey Jaffeux has built a well documented PHP mitt, and has even included instructions for using this code to process inbound emails in CodeIgniter projects. 

NEW Python Mitt:
Inspired by Joffrey’s contribution, Jose Padilla developed a similar project for using Postmark Inbound in Python.

And don’t forget that there is still the Postmark Inbound rails gem developed by Randy Schmidt.

Many thanks to both of them for their contributions!

We are actively looking for your contributions for other languages to add to our developer documentation. If you write a code sample or library for Postmark Inbound, please email us at support@postmarkapp.com. Contributions are appreciated and quality contributions are rewarded with free Postmark credits.

We have a low-traffic API developer email list that you can join as well.

Since our launch on Tuesday, we’ve gotten lots of great questions and comments on Postmark’s new inbound email parsing service. Since, Mark Headd made a great open source contribution that we’ll be adding to our docs: a Node.js listener for Postmark Inbound that stores emails parsed by Postmark in CouchDB.

Even cooler than just open sourcing the code, Mark went above and beyond and recorded a handy screencast to walk you through what he built and how to use it - including the handiness of the way CouchDB handles serving up attachments (watch the video, you’ll see what I mean).

Big thanks to Mark for his contribution, and going the extra mile with the screencast which I’ve embedded below.

If you’d like to earn some free Postmark credits, email me and let me know what you’ve built that can help other people use Postmark Inbound easier! Code samples, screencasts, walkthroughs - the more useful and creative your contribution is, the more credits you’ll get!

Discuss this on HackerNews.

Postmark started by solving the pain of sending emails from your web applications. Today, we’re closing the loop by making it just as easy for everyone to be able to parse incoming emails.

You already know that if your web app sends email, it’s sending it more reliably with Postmark. But wouldn’t it be great if your web app knew how to listen for incoming emails? We thought so too, that’s why we created Postmark Inbound.

Every Postmark server now includes an inbound email address. We’ll turn anything you send to this address into a beautifully formatted JSON object and post it to your application over HTTP, complete with email contents, headers, even attachments.

Postmark inbound is perfect for:

• Replying to in-app messages.
• Analyzing and acting on email content.
• Creating tasks, reminders, and posts.
• Using email to upload files.
• And more!

We’ve worked hard to keep Postmark simple, including how you pay for it. Just pay for what you use, starting at $1.50/1000 emails. The same Postmark credits can be used for outbound and inbound email processing.

Try Postmark Inbound today

Postmark Inbound is available for all Postmark customers and new accounts immediately. Our developer documentation is available to help you get started.

Sign up for your own account and get started right away.

You can also learn how popular issue tracker Done Done and our own hosted version control system Beanstalk use Postmark Inbound every day.

Enjoy!

P.S. Are you as excited about this as we are? Share that excitement on HackerNews!