MPP and Beyond

This is a list of 6 features that you can implement with recent MPP versions that will increase the effectiveness of MPP spam recognition. With our per-feature whitelists you can now exempt CIDR IP addresses from this test.

1) Spam Traps . Populate your site or email posts with easily harvested addresses and configure MPP spam traps to block senders to these addresses. Since our spam traps use regular expression matching you block classes of addresses, such as addresses that begin with numbers, or others that are popular in dictionary attacks. I suggest to examine some of your maillogs to find addresses that are used in dictionary attacks or repetitive spam attacks. This alone will increase spam detection by a few percentage points.

2) Thresholds with Autoblacklists - Define thresholds that suit your environment and let MPP automatically block senders that exceed them. You can define thresholds for hosts that send too much spam or even hosts that send too many clean messages. Thresholds give you 4 parameters to experiment with - number of messages, number of spam messages, time sample period and time in cache. So you can block hosts that send 10 messages and 10 spams in 2 minutes for example. With our per-feature whitelists you can now exempt CIDR IP addresses from this test.

3) Add a second antispam detection module. The combination of Cloudmark and Mailshell or Cloudmark and Commtouch is unbeatable and well worth the investment. Adding SpamAsssassin as a secondary engine, unless you are expert with rules, will only have a negligible benefit.

4) Use header content filtering. With our header content filters you can block email based on country, charachter set, mailer, etc. Building manual rules is not the funnest way to spend your time, but it can increase your detection rates a few percentage points. There are some subjects that are used over and over again and you easily block these as well.

5) Implement Greylisting. If you use Postfix, our Greylisting feature, which is integrated into our policy server, is a great way to cut down spam. Not all users will tolerate the delay, but for those that can, this will block a good percentage of spam - anywhere from 50 to 90%. With our per-feature whitelists you can now exempt CIDR IP addresses from this test.

6) Implement RBL’s in combination with MPP custom spam scoring. Now that MPP has a custom spam scoring algorithm you can accumulate RBL matches and become more adventurous in the RBL sites that you choose to use.

What follows is Message Partners‘ podcast focusing on the Postfix improvements with the release of MPP 3.4. You can either listen to or download the podcast or simply read the entire transcript below.

 

 Standard Podcast [4:48m]: Play Now | Play in Popup | Download

First tell me about MPP’s role with Postfix?

MPP is the only solution that I know of that is commercial or open source that has an integrated policy server and a post-queue content filter.  So we’re the only integrated pre post-queue filter that I am aware of for Postfix.

MPP is innovative in our filtering capabilities for Postfix because not only do we support open source engines like SpamAssassin and ClamAV but we have far better alternatives in the spam and virus filtering engines area by also supporting Cloudmark, Commtouch and many other commercial virus scanners for Postfix.  So it’s a very comprehensive solution.

So the combination of pre and post-queue capabilities makes MPP the premier filtering solution for Postfix in my opinion.

Can you expand on the policy server?

Postfix has a policy protocol where before messages are actually queued they can send all kinds of content information about the message to a server that will decide whether to accept or reject the message.  MPP has made a policy server that can respond to these queries and make intelligent decisions whether to accept or reject messages.

There are many policy servers that are free that are open source but MPP really goes beyond by integrating our policy server with our post-queue content capabilities and with our internal policy engine.  So that means we can very easily enable things like grey-listing on a per-domain basis, we can do spam traps on a per-domain basis, we can do thresholds on a per-domain basis, all types of things on a per-domain or per-group basis that would be very complicated configurations in postfix without MPP.

Why should someone go with MPP and Postfix instead of just going with an open source solution?

Well, MPP is innovative in our filtering capabilities for Postfix because not only do we support open source engines like SpamAssassin and ClamAV but we have far better alternatives in the spam and virus filtering engines by also supporting Cloudmark, Commtouch, and Mailshell and many other commercial virus scanners for Postfix.

The integration of pre and post-queue capabilities and the ability to have separate SMTP policies on a per-domain or per-group basis, something that’s very simple to do with MPP, but very complicated to do with Postfix, unless you’re a really really in-depth Postfix administrator and you’re willing to spend a lot of time writing scripts and learning postfix, we offer a much more simplified solution that is very scalable and has a very good interface that really saves a lot of time.

Unfortunately many administrators are just overburdened and managing postfix or email or routers or any one piece of technology is just a small piece of what they do in a day and anything that can save them time and make their jobs easier is a big benefit for them and MPP delivers on that.

MPP has a new release, 3.4, can you tell me how Postfix fits into that release?

We have enhanced our policy server greatly in MPP 3.4.  Primarily in the area of scalability by adding a dynamic threading model so that we can increase threads as we need them for our policy server.  Before, we have to statically define how many threads the policy server could use similar to postfix, postfix you have to define this statically. However, we found that email is very bursty, and we need to dynamically allocate threads when there’s a spam attack or a big load of messages come in.

So we have created a threading model that will dynamically allocate threads and destroy them when they’re not in use.  This allows you to optimize the memory utilization of your server and respond to attacks very elegantly.

Another area that we’ve increased in 3.4 for the policy server is to add white lists, specifically for certain features.  For instance, we can white list greylisting, or spam traps, or our automatic blacklist thresholds on a per-user or per-domain basis.  So the key areas are dynamic threading for scalability, increase white listing capabilities to fine-tune the environment.

Technorati Tags: postfix, pre-queue filtering, post-queue filtering, threading, scalability, stop spam, end spam, protect email, white listing, SQL, SMTP, black listing, queue, dynamic threading, per-domain, per-group, integrated policy server

Spent the weekend cleaning a computer of Virtumond and tons of other spyware, bots, trojans, backdoors and various crap that an unprotected Windows PC can suck in. Here are a few observations.

First of all, antivirus vendors are really in the dark with spyware. Nod32 happily let Virtumond totally infect the machine along with about 20 other spyware related malware’s. To their defense, the updates were a few weeks old, but the license was valid and Virtumond has been out since 2005, so there’s really no excuse. F-PROT could detect it but couldn’t remove it. They were able to stop it from executing most of the time, but it was still able to execute at times.

Secondly, commercial spyware and spybot programs weren’t a hell of a lot better. Adaware detected Virtumond but couldn’t remove it nor stop it from executing. Adaware did find and remove about 20 other problems, but even after it was finished, Spybot found plenty of issues (free) and SpyHunter by Enigma Software found tons more, even after both of these were done. None of them could remove Virtumond though.

After Adaware, Spybot and Spyhunter finished, F-PROT was able to find some backdoor’s that remained, but still none of the quartet could fix Virtumond. I found a Symantec utility that claimed it could do the trick, but 20 minutes into it’s job I finally found a utility that a laman wrote, Vundufix.exe that did the trick faster than Bill Clinton can stain a dress.

I found it amazing that all of the computer scientists at Frisk, Lavasoft, Enigma, Spybot and Symantec could not do the job that one guy did with his utility that ran in about 20 seconds. There are many conclusions that one could draw - about the power of the Internet that I could find this one guy’s small program or the power of community in the after-days of hyper-investment in the computer security world, but the only conclusion that I draw is that is a great thing to to thank the guy that wrote Vundufix.exe and wish him well. The next time I am cleaning spyware I am going to spend a lot more time learning before I plunk down $40 for software that can’t do what it claims, such as SpyHunter and Adaware and I suggest you all do the same.

Technorati Tags: virtumond, spyware, adaware, spybot, enigma software, lavasoft, spybot

People ask me all the time why there is so much spam and why the problem keeps growing? Why is there so much attention on the spam problem legally, governmentally and technically, yet it all seems to have zero effect on the spam problem? The antispam industry grows at an alarming rate, but though there are no data points to back my claim besides emperical evidence, the spam industry seems to be growing at a far faster rate. My hypothesis of why this condition persists is because spam is a culutural and psychological problem that can not be solved by technology, legislation or government.

Spam works becuase it targets the most primary drivers of human baseness - money, sex and drugs while exploiting our addiction to instantaneous information. Any thing that can i) appeal to the baseness of humanity, ii) do it with some creativity, iii) figure out a good distribution network iv) exploit our addiction to instant information and v) make LOTS of money for the purveyors, will have unlimited success. Wheter it’s a disease, clothing, entertaiment, art - or spam, if it hits on all cylinders there’s no stopping it. It’s ‘the Tipping Point,’ theory and spam fits right in the mold.

The cure for the spam problem faces the same hurdles as cures for drug and alcohol problems, sexually transmitted diseases like AIDS or Herpes or other deadly diseases. Bill Gates money can’t end diseases any more than Mike Katz products can fix the spam problem because they are cultural problems and can’t be fixed with money or technology alone. Of course technology plays a big problem, but email is not just technology, it has become a sociology. Vaccines alone don’t end diseases, it takes Mother’s to bring their kids to the doctors, schools to require vaccinations, governments to require vaccinations, money to create and distribute the vaccines and a healthy profit margin every step of the way. Spam is the same and until there are mulit-level efforts that include education, legislation, technology and cultural wil the problem will persist.

In future posts I will propose fixes, but for now, I have this thought off my chest and I would love to hear your comments.

Technorati Tags: antispam, spam, email security, spammers

Finally, after about 4 months we have realeased MPP v3.4. It seemed like everytime we were about to release some other requirement came up or structural developments became deeper and took longer than anticipated. The result is our best release in my opinion and makes MPP even cooler, more scalable and more useful. I briefly spoke about the features on our mailing list, but here I will go into more detail about what they are and why we did them.

Pleased to announce the we have released MPP 3.4. There has been a lot of work on this release and we have added many new features and structural enhancements.

1) Dynamic threading for Postfix Policy Server - Automatically create and destroy threads based on load.

MPP is multi-threaded for scalability, but we had been tied to a static threading model, where admins had to pre-set the amount of threads that they anticipated using. Creating too few threads can result in resource problems under high load, and too many can lead to extra RAM usage or in extreme cases, hurt performance. The problem is that when MPP is involved in the SMTP transaction we needed a more dynamic threading model. When an SMTP attack occurs, it is important to be able to handle the flood of connections as both good and bad traffic will suffer. When MPP is a Postfix Policy Server, we are involved in the SMTP process, so it was critical that introduced a dynamic threading model to keep up with traffic bursts.

MPP admins now can define a min and max amount of threads along with some information about tear down periods. The dynamic threading model only applies to SMTP traffic and only when we are working with Postfix as a policy server. In order to activate the new queueing model, set to yes.

2) Internal disk queue for quarantine and archival - Remove direct writing to DB under congestion.

In previous versions of MPP the daemon process was writing email directly to database for quarantine or archive. This can effect email traffic if there is a problem writing to the database for some reason. To remove this direct relationship we now write to a disk queue when the database does not respond in time.

3) Hierarchical quarantine/archival - MPP can use esmtp to transport quarantine/archive traffic to a centralized instance of MPP that then writes to DB.

This feature was added because we have many customers with multiple instances of MPP writing to a single database. This creates prioritization issues that can affect traffic flow. To alleviate this we created a new model whereby we have a centralized MPP instance that handles writing to the database while remote instances of MPP transport traffic destined for quarantine/archival traffic via ESMTP to the central instance of MPP. As a byproduct of this feature we can now run 2 instances of MPP on the same server, for testing purposes, and run MPP with no scanners configured.

4) International language support for signatures.

Many MPP customers are outside of the US and want to use our disclaimers/signatures option. MPP could only support US-ASCII charachters in signatures, but now, we can add signatures with other charachter sets. The administrator defines a signature for different charachter sets found in their country and we add our signature only when there is a match.

5) Loading of mppd.conf.xml from stdout, we now provide a script to load mppd.conf.xml from a tftp server and send it to stdout for mppd to use.

Many of our ISP customers do now want to have a web server on their email server, this is currently required to run our GUI module. Now, we provide a script to load our conf file from a remote TFTP server. Our GUI will support editing and saving of config files in multiple locations so now you can have one central config server that all remote MPP’s use.

6) Custom Reject Notices and API - Write custom reject notices or use MPP as standalone scanning service that can be implemented in scripts.

This feature was created because we can not easily support every mailserver in the universe. Plus, some ISP’s wanted to give more verbose reasons why they are rejecting email. With this feature rejection message templates can be created for different reject conditions that can be sent to a remote smtp server or to a script that called MPP. A script can now pass a message to MPP via SMTP and we will apply our policy engine and scan the message and give results back in a template format. The script can parse our results and take appropriate action.

7) 2 new macros for maildir storage and small maildir fixes. Macros are for first/second letter for user part/domain part of mail address: %U1%, %U2%, %D1%, %D2%.

Someone asked for this as they keep their maildirs seperated by alpahabet. It was easy so we did it.

Whitelists by IP for specific features of greylists, spam traps and tresholds/auto-blacklists

This feature was done because when MPP sits behind a relay server we don’t want to some auto-blacklists to the relay host. Our auto-blacklists will kick in if a threshold is exceeded, such as 10 spams in 30 seconds out of 10 emails. A relay server can easily do this and we wouldn’t want to backlist this IP. Once we added one feature we added a few others like grey listing, since local email should not have greylist tests applied, and spam traps, for the same reason as for auto-blacklists.

9) MPP Custom spam scoring - Assign scores to RBL sites and spam engines as an alternative to first-match actions

We added this feature to make MPP more flexible. Since some customers wanted to add scores from RBL sites and spam scanners together we give the flexiblity to do this now. Some RBL sites are unreliable, such as spamcop, and should not be used for rejection criteria, now they aren’t with MPP.

10) McAffee UVScan command line scanner is now supported

Added this for a customer that needed it, but we did it in a cool way. By supporting a daemon like interface we can use McAffee VirusScan utility much faster than other implementations that we are aware of.

11) Added new SQL format
We did a lot of testing and research on a new format for our MySQL quarantine and we came up with a very optimized format for our database and indecies that will scale very well. We took into account domain admin logins and other typical tasks that required hoggish MySQL joins and designed the DB to avoid these issues.

Technorati Tags: antispam, mpp, message partners, msyql, spam, emai security, email compliance, postfix, policy server

What follows is Message Partners‘ podcast focusing on the dramatic improvements in archival and quarantine with the release of MPP 3.4. You can either listen to or download the podcast or simply peruse the entire transcript below.

 

 Standard Podcast [4:09m]: Play Now | Play in Popup | Download

Tell me about Archival and Quarantine for MPP 3.4.

We have added increased scalability features for quarantine and archival by creating a hierarchal system for transporting messages destined for archival or quarantine message store. And also by redesigning our database scheme to scale to much larger organizations.

How will this benefit companies?

Let’s break it down by feature. We’re introduced queuing at three different levels now.

First, whenever a message has to be written to quarantine and archival it used to go directly from MPP to the database but that created problems because if the database was unavailable it was possible that MPP could not process messages and mail-flow could be affected.

So the first thing we did was create an internal disc queue so if the database is not available for some reason MPP writes to disc queue first and then another process writes to the database. So now we can consistently process traffic even during a database failure or, if you have a large message that takes a long time to write to the database we do not interrupt mail-flow.

That’s great for self-contained systems where the mail server and database are on the same server. This queuing will take care of lots of the interruption problems that have occurred in the past.

For our larger installations where there are multiple MPP front ends talking to a single quarantine or archival database we’ve introduced a second level of queuing and something we think is pretty innovative. What we do now is use ESMTP to transport messages from remote MPP instance that just services quarantine. What this allows us to do is transport messages in a standard protocol, SMTP, which has some built in queuing and resiliency, and we can still carry all the attributes of scanning state or wire messages in quarantine or archives in ESMTP headers, so it’s a very elegant and simple way to deal with queuing messages thru a remote archive or quarantine server.

So we have two stages of queuing: first MPP goes to the disc-queue. Then, if we’re going to a remote system, we’re using a process to go from ESMTP to a a centralized MPP that just handles quarantine. Then that MPP instance that just does quarantine or archival has another disc-queue so that if the database connection is lost from that server, mail is still held in the queue and processed when it can be, when the database is available again.

So that’s three different levels of queuing. It’s a lot of resiliency and increases scalability.

Can you tell me more about Archival?

All those changes relate to archival. One of the things we’ve done that enhances both archival and quarantine is introduced a redesigned database scheme. Our old database scheme is fairly scaleable and works for millions of messages but we were relying on some SQL tactics like ‘joins’ which were very very memory intensive. We’ve simplified our database scheme now and we’ve added some fields to make certain repetitive queries like when a domain administrator logs in, for instance, or searching all messages by specific date ranges. We’ve optimized our indexes and redesigned our databases so we do not need ‘joins’ to do these things and that makes the database scale much better, reduces the number of tables, and reduces the complexity of the SQL functionality.

Technorati Tags: quarantine, open source, email security, service providers, scalability, stop spam, end spam, protect email, archival, SQL, ESMTP, email database, queue, disc queue

Please listen to the podcast below, or simply read the transcript. The podcast discusses MPP’s newest release, MPP 3.4, which improves on functionality and scalability, and which makes MPP even better for protecting the email of large email providers.

 

 Standard Podcast [3:38m]: Play Now | Play in Popup | Download

Please describe the newest Message Partners release, MPP 3.4.

This is a release we’ve been working on many months. It’s something we’ve taken a lot of customer feedback on and looked at our competition and decided we needed to focus on functionality and scalability. After a lot of work I’m proud to say that this release has succeeded in really increasing the value that MPP provides and enhanced our scalability and function considerably.

Can you expand on that?

We get into many competitive situations and we deal with all types of customers. Service providers and education space and small to medium business space. Our core customers are really service providers and those customers are increasingly asking for more embedded functionality MPP and they’re asking for better scalability.

So if you look at scalability. Scalability is not just about how many messages per second. It’s really more of a global view of the entire architecture. Not only is it about how many messages can we process in a second but, also, how does our quarantine structure scale, how does our archival system scale, how do our data base structures scale, how does our threading model scale, how do we respond to bursts of traffic because email is a very bursty type of traffic. We have addressed all of these areas in this release.

In terms of embedded functionality. On a very basic level, we function as an integration tool. But an integration tool doesn’t really have enough value for a lot of people to choose us over other products. We need to continually add more embedded functionality into MPP so it really has value as a stand alone product. To that end we have significantly increased functionality in terms of how we can control multiple engines, how we can use comparative spam scoring and our own spam scoring algorithm so we can evaluate many tests. We have added considerable enhancements to our white and black listing methodologies and we’ve done a lot with how we archived messages so we’ve increased the archival capabilities considerably.

How do customers directly benefit from this new release?

Our current customers benefit by seeing increased value in the investment that they’ve already made with us because we have lots of new features and ways to deal with email problems. Our new customers look at MPP and see a much more robust platform and something that can meet very complicated needs or something that can meet very streamlined needs with great efficiencies.

Our service provider customers, which is where the majority of our revenue comes from, will be able to deploy MPP in much larger situations and understand that we’ve dealt with lots of fault areas and improved resiliency in our system, so, increased resiliency and increased scalability.

Our small to medium customers which is really the bulk in the number of customers will see lots of new embedded tools to fight the spam war and lots of new functionality related to archival which has become a very big area for MPP.

Technorati Tags: MPP, Message Partners, Antispam, open source, spamassassin, email security, service providers, scalability, stop spam, end spam

MPP Email Security Now Available on GSA Contract Schedule
June 6, 2007, New Rochelle, New York and Westminster, California - Message Partners is pleased to announce that MPP is now available on a GSA schedule exclusively through Aglow Technologies on GSA Contract# GS-35F-0460T. MPP (Message Processing Platform) has an innovative approach to email security and compliance that blends the best of open source and commercial technologies into comprehensive solutions. MPP specializes in protecting email servers running on Linux and similar operating systems, which is a major initiative in the government and military markets. Aglow Technologies is a woman-owned small business which serves the enterprise and federal markets.

“Aglow Technologies has offered MPP solutions from the beginning, which means they have terrific sales and technical experience with MPP,� say Michael Katz, President of Message Partners. “We believe this partnership will be a great step forward in helping the government deal with their spam and virus problems.�

“Over the years we have gained considerable confidence with MPP,� says Kim Thoo, Director of Sales for Aglow, “We believe our GSA approval will be a tremendous opportunity for us and the government.�

MPP’s comprehensive email security solutions feature best of breed anti-spam and anti virus-technology from market leaders such as Cloudmark, Commtouch, Mailshell, Sophos, Kaspersky, Nod32, F-PROT and others for protecting email servers like Sendmail, Qmail and Postfix.

About Aglow Technologies

Aglow Technologies is a systems integrator that is dedicated to offering best-of-breed security and infrastructure solutions to corporate and government clients. Aglow maintains strong relationships with leading hardware and software vendors such as Juniper Networks (NASD: JNPR) and F5 Networks (NASD: FFIV) and is an IBM (NYSE: IBM) Premier Business Partner and Computer Associates (NYSE: CA) Enterprise Service Provider.

Aglow is a certified SBA Woman-Owned 8(a) and SDB Company.

About Message Partners

Message Partners delivers email security with the groundbreaking MPP, which provides the email accuracy, archival, and per-user and per-domain adaptability that is ideal for businesses and service providers alike. Message Partners supports thousands of customers worldwide and has a global network of resellers, distributors, and partners.