Oct 30 2008
How to test e-mail notifications properly
I frequently get this question about unit or acceptance testing: If sending e-mail or some other sort of notifications is required by the business process, do we test this and how? This arrived again yesterday from a reader, in the following form:
…and one component of our application is email notifications (pretty straightforward stuff). This occurs on registration, forgot password, etc. I am having a tricky time thing about how to test to make sure these email notifications actually arrive. Is it common practice to use the same tool to login to a dummy email account on gmail to make sure the message makes it there?
This seems like a hack, and I’m curious if you’ve ever had to deal with a similiar situation.
First of all, I regard sending e-mail synchronously from the web request as a very bad practice. Talking to an external system can slow down your business processing (and unit testing) and makes the whole process more error-prone as the DNS or MX transport to the external system might time-out. This might cause problems even when the e-mail server is local, since some e-mail servers try to verify the recipient address or server before confirming that the message is accepted. This is especially important if you are working in a transactional context eg you should only send an e-mail if the registration transaction is successful, but you don’t really know that until the whole thing commits. If the database commit fails, you can’t really revoke the message. If you send the message after committing, then there is a chance that the system will die in between and you’ll commit without sending a message.
What I tend to do is use a queue (MSMQ/ActiveMQ/JMS or whatever is available for your platform) and just enqueue the correct message there during the business process. You can then use a mock queue for unit testing and verify that the correct message is queued.
On the other side of the queue, you can have one or more queue processors that pick up the messages and ship them off to the mail server, resend in case of failure etc. That remains a tricky part to test, but you can write an integration test that posts a message to the queue, wait a minute and then check the e-mail mailbox with an automated script. This can run overnight, so you don’t have to run it all the time.
This way you split the functionality into something that is likely to change and should be tested frequently (business process and mail content) and something that is unlikely to change at all after you first write it and can be regression-tested overnight (SMTP message delivery). The first part is nice and quick to test because you can run everything on a developer box with a mock queue and check it from a unit testing tool.
Image credit: Gary Scott
 |
 |
“I regard sending e-mail synchronously from the web request as a very bad practice.”
I think I’ll be using this post as a reference point for people asking for email-sending in web apps.
my IEmailService implementation will always save to reliable queue (usually a simple EmailQueue table in the DB), and a timely-service will pop messages and SMTP them away.
The actual SMTP action along with retry policies, failovers, and such logic, is of no concern to the business application, and has no place within a business process (neither during a web request, or even a smart/fat-client event handler)
as for testing the actual SMTP sender – I’d lower the integration just a bit.
I can test that the queue is “working” in isolation from the SMTP and POP3 actions.
the test will involve calling the actual SMTP_Sender, wait a minute, and then POP3 from the test email account, leaving the queue (pushing and popping) out of the equation
[...] Gojko Adzic ยป How to test e-mail notifications properly [...]
Thanks for this article. We avoid synchronous access to external services by inserting a row into a database during the web request, which is then processed asynchronously by a scheduled task (say every 10 minutes).
I guess this is a different pattern to the queue approach. What are your thoughts on the usage of either one? In my mind the batch processing approach is easier to implement but introduces a time delay (in our case it will be a maximum of 10 minutes before the email is sent). The queue approach seems more complex (requiring a middleware server) but its response time is only limited by how busy the queue is.
Any thoughts, as queue-based systems are not something I’ve used much yet.
Hi
You can config the smtp-server to serialize the message sent to the smtpserver instead of sending the email. Once this is done in your unit tests you can easily read it from the folder and see if it is the expected message.
the config stuff
http://www.singular.co.nz/blog/archive/2007/11/28/using-an-smtp-pickup-directory-delivery-method-for-asp-net-development.aspx
Have faith in the e-mail system and don’t test it.
Eddy.
Ben – your database system *is* a queue, and all queueing systems introduce some latency. But if your batch process runs every 10 minutes and also listens for a “run asap” message, then you can reduce your latency. The best case is everything works and the email goes very quickly, the worst case is that the database transaction commits but the notification is lost (which is the bug being avoided by having the transactional queue) and the user has to wait an average of 5 minutes before the email will actually be sent.
I wouldn’t install an entire message queue application for this purpose alone, and either do the quick and simple thing that you’ve done. But if the message queue system is used in other parts of the system, then I’ld use it directly. Also, the “send mail” interface should hide the database implementation, so if it changes the rest of the application shouldn’t need to be changed.
Eddie – “the email system” might work by magic pixies for all we know, but if the data doesn’t get from the application to “the email system” then the email will not be sent, and you’ll be a doofus with a broken site. And that has to be tested.
Hi Gojko,
have you tried testing email using FiitNesse? Its a total PITA.
I’m not saying its hard to implement, it just that you have to write a test that will sit around waiting for a queue to be processed, and email sent, then received, and then you can begin to make assertions on the email content.
The test has to be robust enough to not hang around forever (contain a timeout), and yet handle a slow queue or even an email that never arrives (the spam filter ate it, honest).
Cheers,
Stuart.
P.S how is the book coming? i’m looking forward to seeing what you have got in the QA section
Hi Stuart,
FitNesse is not good for infrastructural code testing. it is great for acceptance testing. infrastructure is better tested using some code-oriented tool such as JUnit (these tests don’t necessarily have to be included in the basic JUnit test suite, you can separate them in a suite executed overnight). Testing the business part of the process (as suggested by the article) can be done with fitnesse without any waiting or anything. use an in-memory queue and read it directly to verify the contents.
the book is almost done. my copy-editor is looking at it now, and it should be out in late december or early january.
I think i m gonna use this post as a reference too for email testing, and how to go about it, Thanks very clear
Andrea
I have my own mail architecture which i’ve developed over several years so I essentially follow the same kind of pattern. Dump the message into a queue and get back to servicing web clients as quickly as possible. Let the mail queue do it’s job and pick up the messages as and when.
Interesting point regarding if the message ultimately fails to send – supposing there was a typo in the user’s e-mail address. How would the account ever be registered/validated/whatever? I guess we just have to hope that the duplicate e-mail address fields are enough to focus the user’s attention on typing it in correctly!
You could asynchronously the message from the web app in a thread pool thread or similar, but there is a finite limit to the number of threads etc. that can be processed at any one time. At some point the message has to be serialised and processed at a later time.