Mail as Message Queue

Lets first understand, what is Message Queue (Credits: Lovisa Johansson)

A message is the data transported between the sender and the receiver application; it’s essentially a byte array with some headers on top. An example of a message could be something that tells one system to start processing a task, it could contains information about a finished task or just be a plain message.

Message queuing allows applications to communicate by sending messages to each other. The message queue provides a temporary message storage when the destination program is busy or not connected.

The basic architecture of a message queue is simple, there are client applications called producers that create messages and deliver them to the message queue. An other application, called consumer, connect to the queue and get the messages to be processed. Messages placed onto the queue are stored until the consumer retrieves them.

My Requirements

So, while developing my application, I was looking for a service, which acts like a Message Queue, however, my requirements were a bit different, I had some different types of data – Transaction Data, XML Response and some invoice like PDF files which should be generated and stored (for government compliance), all these items cannot be stored on the processing server, due to low disk availability and high risk.

My Environment

I have a server with 2 CPU and 8 cores, limited SSD disk and good bandwidth (5-10gbps, on Amazon EC2)

My Solution

Instead of using traditional Message Queuing applications like Amazon MQ, IBM MQ, I came up with a solution which suited my requirements, it was easy to implement and does not requires extra cents, so I decided to use IMAP/POP3 as my message broker. I had never used any MQ service with PHP, so going with any existing MQ means investing time learning it, I had tight deadlines to meet. The bottleneck was the generation of PDF document which should be generated on the time of transaction, it should be tamper proof (no text / csv files), digitally signed and should be stored for 5 years. (The authorities may ask for any random file and it should be compliant with there requirements). Client did not wanted to store or archive the same on cloud or S3.

What I did?

I installed DoveCot and PostFix on the same box (the box is not connected to Internet), configured the imap/smtp services and created an email account (mq@rajiv.local) with 5GB of inbox size.

In my application, instead of sending the required data and related files to MQ, I sent an email to mq@rajiv.local, I attached the generated files as email attachments, created a timestamp string to identify the message, used different custom email fields to feed in data which will help me to distinguish the email or message and I am done.

The retrieval of messages was really simple, I simply used a script to connect to the server using Imap (143) and process messages one by one, all the variables setup in mail headers and message body were stored in database and the files were moved to the archive server through FTP, I used Cron to execute this script every 180 seconds.

Since everything was happening on localhost, there was little or zero latency, later on, I moved to a dedicated mail server and created different email accounts for different applications or processes.

Benefits of Imap over MQ

There were a few, MQ is used for small messages, my data was around 2mb-15mb, I was not sure MQ supports different data types or Files, but my solution did, also, with the Email ID, I could forward the same message to different email addresses, infact I did (to Gmail :D) for archival and future use.

If you would like to explore my solution or if you think my solution may fit in your requirements, feel free to contact me.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.