MPP and Beyond

What follows is Message Partners‘ podcast focusing on the dramatic improvements in archival and quarantine with the release of MPP 3.4. You can either listen to or download the podcast or simply peruse the entire transcript below.

 

 Standard Podcast [4:09m]: Play Now | Play in Popup | Download

Tell me about Archival and Quarantine for MPP 3.4.

We have added increased scalability features for quarantine and archival by creating a hierarchal system for transporting messages destined for archival or quarantine message store. And also by redesigning our database scheme to scale to much larger organizations.

How will this benefit companies?

Let’s break it down by feature. We’re introduced queuing at three different levels now.

First, whenever a message has to be written to quarantine and archival it used to go directly from MPP to the database but that created problems because if the database was unavailable it was possible that MPP could not process messages and mail-flow could be affected.

So the first thing we did was create an internal disc queue so if the database is not available for some reason MPP writes to disc queue first and then another process writes to the database. So now we can consistently process traffic even during a database failure or, if you have a large message that takes a long time to write to the database we do not interrupt mail-flow.

That’s great for self-contained systems where the mail server and database are on the same server. This queuing will take care of lots of the interruption problems that have occurred in the past.

For our larger installations where there are multiple MPP front ends talking to a single quarantine or archival database we’ve introduced a second level of queuing and something we think is pretty innovative. What we do now is use ESMTP to transport messages from remote MPP instance that just services quarantine. What this allows us to do is transport messages in a standard protocol, SMTP, which has some built in queuing and resiliency, and we can still carry all the attributes of scanning state or wire messages in quarantine or archives in ESMTP headers, so it’s a very elegant and simple way to deal with queuing messages thru a remote archive or quarantine server.

So we have two stages of queuing: first MPP goes to the disc-queue. Then, if we’re going to a remote system, we’re using a process to go from ESMTP to a a centralized MPP that just handles quarantine. Then that MPP instance that just does quarantine or archival has another disc-queue so that if the database connection is lost from that server, mail is still held in the queue and processed when it can be, when the database is available again.

So that’s three different levels of queuing. It’s a lot of resiliency and increases scalability.

Can you tell me more about Archival?

All those changes relate to archival. One of the things we’ve done that enhances both archival and quarantine is introduced a redesigned database scheme. Our old database scheme is fairly scaleable and works for millions of messages but we were relying on some SQL tactics like ‘joins’ which were very very memory intensive. We’ve simplified our database scheme now and we’ve added some fields to make certain repetitive queries like when a domain administrator logs in, for instance, or searching all messages by specific date ranges. We’ve optimized our indexes and redesigned our databases so we do not need ‘joins’ to do these things and that makes the database scale much better, reduces the number of tables, and reduces the complexity of the SQL functionality.

Technorati Tags: quarantine, open source, email security, service providers, scalability, stop spam, end spam, protect email, archival, SQL, ESMTP, email database, queue, disc queue