Search AmavisD and Postfix Mailing Lists with MPP!
I am pleased to announce a free service to search the popular MySQL, AmavisD and Postfix mailing lists using the MPP email archive viewer. To give it a try logon here:
http://herbie.raeinternet.com:20000
user: mknews
pass: search
Go to archive using the link on the top and search away.
The end-user view of Antispam
Today I read an article in the Wall Street Journal by Lee Gomes that gives an excellent view of the end-users perspective of the antispam fight. One of the biggest topics of discussions that I have with administrators of antispam solutions is the question of how to deal with spam. Should they quarantine, mark subjects, discard, reject, etc. It is a religious debate that has more to do with company policy, personal preference and IT budgets than anything technical, but it is interesting to me to hear Mr. Gomes prove how smart he is by agreeing with my general advice to people :-). My advice to all admins is taht users don’t care so much about getting extra spam but the minute you quarantine an important email you are dead meat.
With that precept in mind here are some general guidelines that I recommend.
1) Make sure that your antispam solution can fit everyone in terms of spam actions but start with the least restrictive policy. The least restrictive policy is either putting spam into a spam folder that is accessible by the users email client or marking subjects.
2) Only quarantine spam if your boss wants you to. Spam quarantine is expensive to manage and rarely checked. You will get a screaming call about a lost email and most likely the user has no idea how to check their quarantine and you will be the one clamoring around looking for the email and you will certainly have your head chewed off for being the sole cause of the malady. If you want to quarantine for all people make sure that users or admins can turn this off for themselves, as they can with MPP.
3) Mr. Gomes points to his experiments with changing filter levels from more restrictive to least restrictive and being disappointed with the results. My experience is that these knobs are mostly useless, especially the ones that make futile attempts to categorize spam. Give me the Viagra and Vicodin spam, but I sure don’t want to see the mortgage offers. Who thinks like that? In my experience spam companies to a horrible job of categorizing spam and it seems like a big waste of time anyway. Give your users the ability to change spam actions but these extra knobs will only bring problems as they did for Mr. Gomes.
4) Perfection is as elusive as the $40million that Barrister Hollings offered me from his hideout in Kenya. He points out that even the great Postini, which Google handsomely overpaid for, has false positives in the high sensitivity settings and misses lots when the sensitivity is set down. Don’t expect perfection and don’t back yourself into a corner by over promising and quarantine or rejecting too much spam.
5) If you decide to use quarantine systems less than 5% of your users will ever logon and of those 5% probably only a handful will ever dive too much around in the settings to see what’s available. Beware of a strategy relies on end-users learning your systems, they will mostly ignore them until it’s time to take off your head.
Backscatter Problems Persist
Backscatter problems are persisting and seem to be the new scourge of spam. They expose a lot of weakness in antispam filtering algorithms and require new approaches to truly stop them. Here’s why:
- Many backscatter attacks do not contain any message content, only bounce notifications. Stopping all bounce notifications or messages from mailer-daemon is dangerous
- Many backscatter attacks come from legitimate email servers so you can block the offending servers at the SMTP level without blocking legit email.
- Backscatter attacks can be extremely high volume and can crush a spamassassin/amavisd installation that is not micromanaged.
At Message Partners we have been working on some approaches to backscatter problems for our customers. The most heavy handed approach, and one that I have been utilizing on our site, is to block all traffic that originates from Mailer-Daemon with this regular expression for header filters: ^From:.*Mailer.*$ . This is annoying because you can’t see legit bounces, but I have taken an analogy from the US Post Office. If you need acknowledged delivery then you must send with delivery notification service. If you send a letter to the wrong address it may take weeks before it comes back to you, if it comes back at all. Anything I really care about I send with some delivery notification. I have decided that this should apply to email as well. If I really care about the email then I should send it with some delivery confirmation, hence, I have rationalized filtering out all bounces. This has ended my backscatter problem for the most part, but many messages do not come from mailer-daemon so the problem still exists, but on a much smaller scale.
We have a more intelligent solution in beta that utilizes our auto black list feature. With this approach a threshold is set that if x amount of messages are received that violate a content rule then all messages from any IP that sends with the same subject will be temporarily blocked. For example, if gmail.com sends me 400 bounces in 2 minutes (which they’ve done), and they all have Undeliverable Mail in the subject, then any message with Undeliverable Mail will be blocked during the block period. This is working pretty well, but not a complete solution by any means.
Some of our engine partners are quickly improving their backscatter detection, but when there is no actual message content they all seem to stumble.
Conclusion — Backscatter remains a problem but MPP is helping with our customers with a solution. In the meantime, administrators should be really careful about the rejection notices that they sent.
Technorati Tags: backscatter, spam, email security, mpp
Backscatter Sucks
Yesterday seemed to be backscatter day for us and for a few of our customers so I thought I would write a little bit about it. First of all, a brief definition. Backscatter refers to bounce messages that come from spam that was bounced by some remote site. Hopefully, it stems from someone forging your email address as the original sender. The alternative is that some machine in your network was exploited, but mostly it is just simple forgery. The net result in either case is that you are flooded with bounce messages from all over the world from email that was sent to unknown email accounts or recognized as spam before it was accepted.
Backscatter is a major pain in the ass for many reasons. If your email server bounces the bounce messages, for example if the original bounces are going to unknown users, and the rate of bounces is high then your mailserver can be blacklisted as a spammer. For example, forged mail to stevewashedupbillionairecase@aol.com from wannabebillionaire@messagepartners.com is bounced by aol.com. Now message partners bounces wannabebillionaire@messagepartners.com since it is unknown. If you multiply this by 1000 as can happen in a backscatter deluge then you can imagine that AOL get’s pissed about all of our bounce messages and they will blacklist us for a while
Even more annoying is when the forged address is your real email address. Now your mobile email device is going nuts telling you about all of the bounce messages that were sent to invalid users in your name.
Here is an example of headers from backscatter:
Microsoft Mail Internet Headers Version 2.0
Received: from 68-185-134-174.dhcp.jcsn.tn.charter.com ([68.185.134.174]) by ppcwm.org with Microsoft SMTPSVC(6.0.3790.3959);
Wed, 26 Mar 2008 18:37:03 -0400
Message-ID: <000501c88f92$0569e9fb$dcc2daaf@ilefylfs>
From: “hubert chanshin”
To:
Subject: Massive discounts for purchases in the month of March are available.
The problem is further exacerbated because most filters don’t bounce the message, the send a bounce message, hence spam gateways see the messages as legit bounces and don’t classify them as spam.
The problem is because of a major design flaw in the SMTP protocol that allows anyone to say that they are anyone when sending email. There have been a number of bandaids to fix this such as Sender Policy Framework, but they are mostly ineffective. I get a laugh looking at all of the bounce messages I see from the recent backscatter attack, many have SPF checks that show a violation, yet they still send me a bounce.
You can learn a lot from all of these bounce messages and to me, being geekish in the antispam industry, I find it interesting. I can see what types of email gateways are in use, how many people use home grown spamassassin based gateways, how different spam configurations are setup, how botnets work and more. But most of all, backscatter is just annoying.
So here are some tips to end it.
1) With MPP we were able to block many thousands of messages using our thresholds. Once we detect a certain amount of bounces from a remote server in a period of time we temporarily block the host.
2) With MPP we also blocked many using our contentfilter module and setting some very basic PCRE based filters to block based on content. This is not very dynamic, but pretty effective.
3) The Postfix site has some good tips, but it is a similar approach to using static content filters as described in step 2.
So in summary, backscatter sucks, MPP can help, but it is a problem endemic to a poorly designed sender verification protocol (i.e. none) in basic SMTP.
Old friends - MPP and SpamAssassin
Despite the fact that all MPP antispam plug-ins out-perform and are far more efficient than SpamAssassin, many MPP users still use SpamAssassin in their antispam arsenal. MPP has many advantages for use with SpamAssassin and I thought I would articulate a few in this blog post.
- Quarantine Management - MPP has an excellent quarantine and archive manager that can be customized and translated and adapted for any application. MPP quarantine management for SpamAssassin is great for service providers because it is multi-domain and supports many different authentication schemes.
- Amavisd replacement - Amavisd is great for super-power users, but for the rest of us it can be complicated, slow and resource hungry. MPP is easy to manage and deploy and offers many features and options not available in AmavisD including GUI management, Archival, Attachment Stripping and much more. MPP is much faster than AmavisD, and even though SpamAssassin is still a horrid performer, MPP will make SA feel faster.
- Scaling SpamAssassin - MPP has native load balancing algorithms to load balance across multiple instances of spamd, either remote or locally.
- Layered spam protection- MPP effortlessly combines commercial spam scanners with SpamAssassin.
- Familiar options like score reports, header/subject marks, spam symbols and more
So here are just a few of the many ways that MPP makes SpamAssassin easier to use, easier to manage and easier to integrate. MPP has been working with SpamAssassin from the beginning, like an old friend who’s sort of a pain in the ass, but is a cheap date, pretty fat and always a good laugh.
Technorati Tags: spamassassin, email security, antispam
Think Microsoft is Bad? You’ve seen nothing…
I have been tossing this idea around for a while, boring whoever would listen to me and scaring people who think I may be a little nuts. I have tried to write a blog post about it but I have not found the right context until today. The fundamental idea is that if you think that the MSFT or long forgotten IBM monopolies were bad for competition in computing, you haven’t seen anything until you see the beast that Google could be. Without severe vigilance by anti-trust forces or radical changes in market dynamics Google is set to make any other monopoly in computing look like a child’s board game.
So first, the context. The context is an article on the front page of the Wall Street Journal today (Feb 25, 2008) regarding European antitrust chief Neelie Kroes’ attitude towards Microsoft’s promise of sharing more information with rivals. Her assertion, according to the article, is that we have seen these promises from MSFT before, but nothing really comes from them. The EU is not going to back down until there are real changes in how MSFT shares information with rivals. My thought is when is Ms. Kores going to start thinking about Google?
Google controls how more people view the Internet than anyone in the world. Google nearly processes more email, both personal and business, than anyone in the world. Google serves more Internet video than any single entity in the world. Basically Google controls the content of the Internet and if you aren’t good to Google, you are going to be buried in Internet anonymity. If you pay to play then you will be rewarded with lots of traffic, but at a high cost.
In the old days you could simply create good, relevant content and Google would rank you pretty high in searches That era went away quickly though and the new method to drive traffic was to feed Google was with bogus landing pages and other SEO tricks. This soon changed and now the main way to drive traffic is via buying or displaying Adwords. Of course, a hot story, good buzz, good PR, good sex, good pictures, a good joke, etc. can always spike traffic, but in terms of permanent trending, buying adwords has become an essential tool to every web master and if you stop paying Google, your traffic will plummet. I see this effect regularly on our site, http://messagepartners.com, when our adword subscriptions pause.
On the surface that sounds like simple good business for Google. But if you dig even a little you will see that it is far more insidious than it seems. Fraud is heartily supported by Google and click farms and search landing pages drive the price of adwords sky high while punishing legit businesses that can’t compete with the click farmers. Try a site like http://citinbank.com/ or http://spamware.net/ and you’ll see what I mean. These sites have zero informational value, they are just click farms that trick unsuspecting users. Google loves them though, they drive up legit businesses ad costs through the roof while making money for Google and the site operator for benign clicks. It is hard for an average person to see that these sites are causing such problems because it doesn’t cost the surfer anything to click on a link, but those clicks cost legit businesses plenty and put huge amounts of money in Google’s pocket. Our adwords for messagepartners.com that used to be in the range of .25 to $1.20 a click are now well over $5.00, far out of our league and far more than they are worth for our type of site in my view. It is my view that these prices are driven largely by click fraud or artificial auctions on Google’s part, but since adwords are blind auctions, there is no way that I can ever know who I am bidding against. Message Partners is one small business that can not afford adwords any longer, but when you multiply this effect by the hundreds of thousands of small businesses who see their costs sky rocket from Google sponsored fraud and secrecy you can see the damage that this causes. The supposedly level playing field that the information age should bring is really just a rehash of the same old pay-to-play business models that were offered in TV, Radio and other media. If Google is controlling the flow of information to the highest bidder then the payoff of the Information age for consumers is questionable.
I am not aware of studies that show how search rankings fall when adword dollars stop, but they are certain to come. It is only a matter of time until some business with enough money and interest will capture the interest of regulatory bodies to make Google open up their search and adword price algorithms to show that high paying, politically correct, or Google favored sites aren’t given preferential treatment under the guise of neutrality. Google takes the approach of the benevolent wizard behind the curtain driving all of this web traffic, but with all of the billionaires that Google created, including individuals, Stanford University and VC firms, are we really suppose to totally trust Google and believe that benevolence and not immense greed is driving Google?
Beyond web surfing, Google has fast moved into the email space. In five years they moved from a techy niche player to a major force in both consumer and business email. Between GMAIL and their Postini acquisition Google is handling a sizable percentage of global email traffic. They found an insidious way to monetize personal email with adwords related to your personal communication content and who knows what their plans are for their business services to capitalize on the content of your email? My guess is optionally eliminating subscription fees and putting content based ads in the signatures of emails. While it is easy to draw parallels between Hotmail and Yahoo, neither has found a way to monetize your email content in such an insidious way as Google does by actually analyzing your emails content. Adding signatures to emails as MSFT does is far more transparent than all of the data mining and monetization that Google does with your personal email. If Google does start with my theorized content based signatures then the privacy invasion and monetization of your internet habits have only continued. On another not it is worth noting that Google is already behaving in an anti-competitive way with free email hosting for small businesses so the seeds are definitely there for foul play on Google’s part.
The video space is slightly more difficult for Google because media companies seem to be far better funded and aware of the danger that Google’s information monopoly poses to their business models and to copyright holders. Google is not finding it nearly as easy to monetize on video content and copyright laws are serving their purposes of protecting the rights of owners.
Of course the billionaires of Google want copyright laws thrown away, but it is a tough sell. Of course they want to put all of the worlds libraries in their search engines, but why should so many authors toss away their rights so Google can monetize their work?
So to sum up my view, if one company controls what you surf, what you see, what you hear and what you send on the Internet then there is a large potential for abuse. This is information control on a scale never before seen and if this company is motivated by greed under the guise of science or if it unintentionally makes bad choices there can be very severe costs to consumers, businesses and the flow and sharing of information. If Google decides what you see, are we really better off than the pre-Internet days?
Technorati Tags: google, antitrust
Another Email Archival Acquisition
Dell has bought MessageOne, which is a big affirmation of the fact that the email archival as a service market is really heating up. So many regulations around the globe are requiring email archival and it is ideally suited for SaaS models, especially when combined with legal discovery, mailbox restoration and other services. MPP is perfectly positioned for this trend as it is the ideal technologically neutral solution for email providers to use in the basis of their email filtering services.
Zimbra…..by Microsoft?
If Microsoft were to acquire Yahoo, what would happen to Zimbra? Anyone’s guess is welcome. If Zimbra survived it would be a huge step for MSFT as they would be supporting the Linux based collaboration market. The Exchange replacement market is gaining steam, though I don’t know if MSFT feels this or not. If they killed Zimbra they would anger a lot of people in the Linux community as well as some pretty big customers. I’m sure that Cox and others who made big bets on Zimbra would be very angry if MSFT shut that product down. My guess is that they will spin it off and avoid the whole headache. I would love to hear other’s thoughts.
Technorati Tags: microsoft, yahoo, MSFT, YHOO, zimbra, linux, microsoft yahoo acquisition
Trend Micro vs. Barracuda Explained
It seems that most of the articles I read about the Trend vs. Barracuda suit miss or obscure the point so here is my attempt to explain it. Ironically, I discovered Trend’s patent a few years ago and showed it to our IP attorney who agreed that it is ridiculous and that most security vendors are exposed on this issue. Simply put, Trend has a ridiculous patent on the concept of scanning for viruses at the gateway - whether it be for email, FTP, www, etc. It is a broad system patent and should never have been granted in the first place.
Trend is suing Barracuda for the simple idea of scanning for viruses at the gateway. Since Barracuda uses ClamAV, owned by SourceFire, and ClamAV probably doesn’t have deep enough pockets to be interesting to sue, Trend sued Barracuda. Most commercial AV vendors have paid Trend to remove their liability but ClamAV never did. Plus, Barracuda has pissed off a lot of companies (i.e. Trend) for wrecking the low end of the antispam business with Walmart style pricing and a crap product, but that is a different story.
In any case, the central issue is the ridiculous gateway antivirus patent that Trend has. However, Barracuda seems to be in a race to be even more pathetic by trying to turn this into an open source issue with Barracuda as the great defenders of open source. That is a big joke. Barracuda is plenty rich, they have plenty of lawyers and they made a calculated risk to eek a few more dollars out with open source rather than paying some AV vendor a small per-box royalty. They wanted a disruption of the low end, they paid for the marketing to make it happen and they got it, but this is nobody’s issue but Barracuda.
Of course there is the larger issue of software patents but that is a fight far bigger than Trend and Barracuda and far beyond the scope of this rant. I say tough luck to Barracuda and thank goodness that we are too small to be on any radar screen for lawsuits.
Technorati Tags: trend micro, barracuda, antispam, email security, clamav, open source
Another Payday for Open Source
The last 18 months has seen mega open source projects cashing in like crazy. Now it’s MySQL for 1 Billion. Who could believe it? It’s a great payday for their VC’s and the insiders, but is it good for the peons? Time will tell. All of the major projects related to MPP’s little view of the world are cashing out nicely - ClamAV, Zimbra, MySQL, etc. But who’s next? SpamAssassin was gone long ago, AmavisD is owned by a huge government backed research organization, Postfix is owned by IBM and the other projects that effect us are too broad to be sold, such as Boost. But hell, if you can get 1billion for these projects I can’t imagine that even IBM or the government of Slovenia would snub their noses at the idea of selling. There are no sacred cows in the open source business.
All this activity reverts to old ideas that I have blogged about - open source is a business model for the mega rich. The concept that there is a nice genius in his living room somewhere writing code that we can all use for free is a long dead myth. Certainly many great open source projects are free and help us all avoid buying software while solving problems, but this model is dying faster than HD DVD.
MySQL, company CEO Marten Mickos says that the deal is a huge win for open source. I ask, when will Benchmark Capital open source their bank account? I’ll be the first to download that!
Technorati Tags: mysql, sun, mpp, penthouse centerfold