Email Testing Flow As It Should Be

About The Author

Andriy is a Growth Manager at Mailtrap, a product that helps people inspect and debug emails before sending them to real users. He has over 5 years of …

More about

Andriy
↬

We spend a lot of time and effort crafting emails with a specific purpose: to make their recipients read them and take the desired actions. The three known bottlenecks of every email sequence include:

How can we address these challenges? It is recommended to follow rules and best practices of building and sending emails.

But how do we know that they work?

By testing every single email aspect! Unfortunately, email testing is often underestimated and leads to email testing mistakes that kill all your effort spent on creating a great email sequence.

Let’s talk email testing! In this article, we’ll explain how a proper email testing workflow can help you improve email sending efficiency. We’ll describe common testing approaches and mistakes, and demystify the seamless email testing flow.

With this article, you will enhance your testing workflows by covering all the important aspects, and saving time and stress with suitable email testing methods and tools.

Are You Keeping On Top Of Your Email Metrics?

While there’s always something to be improved, it’s important to understand when you underperform and need to take action right away.

With any type of email you send, you need to track at least these metrics:

Open rate	How many messages you sent vs. how many of them were opened.
Bounce rate	How many emails were returned to you?
Click-through rate	For those emails that contain links, how many links were clicked?
Unsubscribe rate	For marketing email campaigns, the unsubscribe option is required.

When sending marketing messages (such as newsletters, special offers, abandoned cart emails, and so on), you can compare your rates to the industry standards. However, they will also change over time and in different circumstances.

According to Hubspot, the average email rates are:

The research conducted by Constant Contact shows the following numbers:

With transactional emails, it’s a bit trickier, because the open rate you should strive for depends on the type of email you send. For example, reset password emails should be opened by the majority of your recipients, let’s say up to 90%. Order confirmation emails won’t have such a high open-rate but will receive much more interest than a marketing campaign, even with an exclusive offer.

The other side of the coin for transactional emails is that they should be opened by the right people at the right time. Otherwise, such emails are useless or even harmful. Imagine that John has just signed up for a financial service to create a tax report. To proceed, he needs to click a button in the confirmation email. After three hours, when John’s working day has already finished, he receives an email welcoming someone named Jack. Will he move on with that service? It’s doubtful.

We are pretty sure that you test your emails before sending them to avoid such ridiculous situations. But are you certain that your email testing flow is comprehensive and smooth enough?

Let’s get to the bottom of TypeScript, and how to use it well. In his upcoming online workshop The TypeScript Masterclass, Stefan Baumgartner will dive into type systems and how to use them well when writing JavaScript. Online, and live. August 5–19, 2021.

What Should You Test For Different Sorts Of Emails?

We have already mentioned the difference between marketing and transactional emails. Their purpose, sending method, and performance vary, and so should the testing flow.

However, the goals of testing are common for all types of email sequences — ensure deliverability, content excellence, and engagement. That’s why there is a list of aspects that you should test for any kind of email.

Universal Email Testing Aspects

Even when you use a dedicated email marketing service, you need to check whether all integrations work perfectly fine, especially, when it is first set up.

Email template.
No matter what the message’s purpose is, it should be correct and visually appealing for every recipient.

Email headers and subject.
The sender’s name and the email subject line are the first two things the recipient sees and considers when clicking your message to open it or to route it to a spam folder. Add more focus here!

Spam checks.
Would you like to have a list of precise criteria that cause emails to be marked as spam? Everyone would, especially spammers. That’s why you can only follow some general rules and use dedicated services that analyze the reputation of your sending IP, the content of your email headers, the correctness of your HTML, whether you use certain phrases, and so on. Transactional emails can get classified as spam, too.

Peculiarities Of Transactional Email Testing

When you send triggered emails, especially from your own application or online service, you should take care of a few more aspects.

Test your sending script. Here are two important aspects:

If you are launching a large-scale system, it’s important to perform email app load testing. What if 2,000 users simultaneously click “reset password”? What happens next? How soon will they receive an email confirmation?

Marketing Emails Testing Focus

When you use a bulk email service, in most cases, you have built-in automation and basic testing functionality.

However, there are a couple of moments that require manual checking. They may seem obvious, but are still often overlooked.

This email failure quickly became popular on Twitter. Dozens of cat memes on social media are obviously funny, but this is not always the type of PR activity you expected to be part of.
Take care of the proper naming for your clients’ lists — in some tools, they can be visible for your customers when they double opt-in or unsubscribe.

Common Email Testing Mistakes

Everyone has received an email sent by mistake at least once. Wrong links sent in promo campaigns don’t cause much harm, but what if your broken email script can lead to a data breach or a system crash? With strict data privacy guidelines, such as GDPR and CCPA, unintentional emailing to unsubscribed users or using email addresses of people who have never subscribed to your communication can lead to conflicts or penalties.

There are two main types of email testing mistakes:

In many teams, the email testing process is limited to sending a few test emails to your own or colleagues’ inboxes. Alternatively, public email inboxes for testing are used not to flood personal inboxes.

What can such an approach inform you about? Well, you can learn that your email sending service is functioning and that your email content is displayed correctly in your recipients’ email clients (or in a web browser). You can also ask your colleagues to click links and read the email copy. But what about other email clients and, even more importantly, deliverability?

Another point is testing 5 emails out of 5,000. When you have a robust email-related system, you have to make sure that it functions correctly. We have mentioned the importance of merging mechanisms, so here’s the case — can you be certain that the billing information of Mr. Bluesky & Co. doesn’t go to Bluebird & Bluebird because of a tiny mistake in your database? The most efficient way is to run a set of automated tests (for example, with Selenium) and check the content of each email.

Correct Email Testing Process

A smooth email testing process is based on simple principles:

Email Testing Toolkit

There is no ubiquitous email testing flow because the process of testing depends on the type of email campaign you send, the method you use (whether it’s an email marketing service or your custom email functionality in your application), and the team you work with.

Development and QA teams who build transactional email sequences usually run automated tests by integrating testing frameworks and tools, such as Selenium.

For marketing campaigns, it’s more common to create an email testing checklist. I’d recommend creating a custom email testing list based on the aspects explained in the “What should you test for different sorts of emails?” section and competing it with the tools of your choice.

When you select an email tool, especially a paid one, pay attention to those services that offer wide functionality and can be used by both marketing and development teams.

We have talked about the general rules of successful and painless email testing. Now let’s go into detail about each type of test, with examples of tools.

How To Test Sending Infrastructure

This type of testing is usually used by developers, but there are tools that can come in handy for marketers as well.

In general, you should test your mail server when you establish a new integration, change your setup, or maintain your own email sending infrastructure.

Telnet is a computer protocol that provides communication with a remote server. It’s also a command-line utility available in most common computer systems, including Windows, Mac, Linux, and more. It can help you test server connection, ports open for email communication, supported SMTP commands, relaying specific email addresses or domains.

To use this utility for testing, you will need to run a set of SMTP commands in a telnet client (which is pre-installed on the majority of systems).

Wormly is an uptime monitoring service that also has an SMTP/Mail Server Test. It can help you test whether your SMTP server is configured correctly by sending a test message to your email server. Wormly will log the SMTP conversation for you so you can check and debug errors or exceptions if there are any to be found.

It’s super simple to use: you just need the address of your sending server, a recipient email address, and a port. You can also share the results of your team with a shareable link. Here is the case of how a marketer can use it — run a test in Wormly, proceed if no errors were thrown, and, if any were found, send a link to your dev team for troubleshooting help.

Wormly can be also useful for SMTP server monitoring, which I’ve mentioned in the Correct Email Testing Process section.

Gmass is an email marketing service inside Gmail. It also has a mail merge functionality and cold email sending options. The tool is a paid service, but it has a few helpful free tests:

If you expect high loads on your email server, it will also be good to perform load testing with a tool like Apache JMeter™ or an email sandbox service.

How To Test Email Content

In this category, you have the widest choice of tools. We will name a few to provide you with a starting kit.

Email Template Checkers

When building an HTML email template with a special template builder or using a drag-and-drop editor in your email sending tool, you will have a preview option. It will display how your template is rendered and, in most cases, how it looks on different devices.

If you need a separate solution for HTML preview, you can use a tool like PilotMail. It provides an email builder, layout viewer, and a test email sending to 10 addresses. There is a free plan, but the paid one costs just $3/month.

With no rendering standards for email clients, it’s important to perform email client testing to make sure that your email will look perfect in any of the recipient’s email clients. The most popular tools in this category are Litmus and Email on Acid. They generate previews for a list of email clients and allow you to manually compare them and look for issues. You can experiment with both of them on a 7-day free trial, while further usage starts at $73/month.

Another approach for email client testing is used in HTML Email Check and Mailtrap (As a friendly disclaimer, I actually work there.) These tools analyze your email template for HTML and CSS validity and provide you with an actionable list of errors for each email client. Free plans are available for both tools, and their paid subscription plans start at $10/month.

Email Content Checks

Spell checkers and content testing tools are helpful for emails as well. Grammarly or ProWritingAid can help you check your email copy for errors and typos (both have free plans). Hemingway Editor can give you tips on improving readability and if you’re struggling with writing, Conversion.ai can actually write you an email copy.

The Subject Line And Preheader

Email marketing services usually provide options to preview the subject line and preheader. You can also play with TESTSUBJECT by Zurb that displays email subject and preheader previews for a few types of mobile devices. Preheader Testing Tool also provides similar functionality.

Subject Line Tester by CoSchedule and Send Check It can help you optimize your subject line for driving more opens.

How To Test Email Deliverability

Finally, we’ve made it to our favorite, but the most complicated, point — email deliverability. A whole set of factors impact your spam score: sender reputation, authentication, and email content.

There are a few tools that test your email against all the criteria.

You can start with Mail Tester. It’s free, but it provides a number of important tests. You can view your message and its source code after sending your email to a provided test address, get a SpamAssassin score, authentication check, message body analysis (it also shows HTML size), blacklisting report, link validation, and more.

Free Spamcheck by Postmark offers similar functionality. You can use it without sending a test message — just paste your HTML code with all headers and get an instant result. You can also integrate Spamcheck with your app via their JSON API and run automatic spam checks for all emails you send.

If you need a more robust solution, there is GlockApps. It has a 14-day free trial and then offers subscription plans starting at $79/month. By the way, their website states that:

GlockApps is known for its powerful reports available for registered users. It provides inbox placement, reputation check, DMARC analytics, bounce analytics, automatic tests, monitoring and alerts, and template editor. In addition, there are a few useful separate free tools: domain checker, inbox email tester, inbox insight, DMARC analytics, and uptime monitor.

Deliverability tests are also included in many full-featured email sending and testing tools.

Summary And Main Takeaways

The main purpose of this article was to prompt you to take email testing seriously, run it as a separate project, and pay attention to every important aspect. Email is not just another line in someone’s inbox — it’s a part of the user experience of your app, website or blog. It’s ideal to be able to have a team of marketers, developers, product managers, and QAs who can work together on spotless email sequences. But even if you are running a small project or have limited time and resources, you have a great list of handy tools that can do the job for you.

For any type and scale of the email-related project, it is worth allocating time to establish an email testing workflow and to experiment with tools. This will save you time and improve the quality of your campaigns at every iteration. And of course, you are very welcome to share your email testing stories and approaches in the comments!

Related Reading on SmashingMag: