Really neat way to stop spam ( and sort any email )

[silver]

Yesterday I got 10 or 20 spam / UCE emails all with the same title but from different email addresses, normally I just hit the delete w/o opening or downloading them (using popcorn - which rocks!).,.

I have seen peeps mention mailwasher so I thought I'd give it a look,, here is my view on what it does and some of the drawbacks (as I see them):

mailwasher behaves like a pop client, it connects to the server and tries to work out what is spam / junk and any it spots it flags as potential spam, you then have a choice delete any emails before they are fully downloaded and / or send a ( fake ) bounce message, not bad...

Going by the faq here and from information given by users (blaming Gem here!) it appears that the latest free version 2.0.40 is limited to one account,, this was apparently not the case for the free version in earlier releases (!)

Mailwasher works out what is spam or junk by using various open relay lists (which you can add to) and some keywords which are kept in a file (which you can also add to). I have certain reservations on this technique. Open relay lists are nice but they do not always have all open relays listed. Unlikely perhaps but you could get valid email sent from an open relay. Not all spam comes via open relays, so in this case you are left with 'key word' matching. The keyword file does not sound flexible enough, I don't think it had a 'good word' list (it may do) which might help make it more accurate. Are people really going to go edit a file to put in a new keyword they have thought up, perhaps but not everyone.

There would appear to be a 'race condition'; if you have mailwasher and your normal pop client running, both could try to connect at the same time, or it's possible that your email client will download spam or other unwanted email before mailwasher has had it chance to deal with it. I would guess that most users turn off automatic email retrival for their pop client and manually download the email once mailwasher has run (note this does not eliminate the 'race condition' but makes it less likely).

Presuming that mailwasher must at least download some of the email to work out if it's spam you are downloading it twice (though it has to be said that mailwasher probably only downloads a small amount from each message).. This is also a benefit since you can decide to delete the mail from the server w/o needing to download the whole thing (tho this is just as easy using something like 'popcorn' perhaps?).

Only runs under window (or linux if you install wine!).

Some manual intervention appears necessary whenever you want mailwasher to 'do something', it does not appear to operate silently in the background. You must use it's GUI each time in order to made it delete or bounce emails.

Mailwasher may do everything you want and it will do it's thing with very little need for any sort of setting up, it's also free so I think it may well be usable for what most people want.

..

Which brings me onto what I think is a much better and more powerful way of doing email filtering..

( I run a mailserver in my home network, it collects mail from 5 different accounts every 10 minutes, I also have an incoming server that listens for smtp which I do a pop fetch from. Email is then stored locally and I connect my local mailserver rather than connecting out to the internet. )

I had heard of using 'Bayesian Statistics' to determine the likelyhood that a particular email is spam or not spam. It seemed there must be a pop proxy that allowed this type of sorting.. a quick search and I found popfile

Popfile basically sits between your email client and the popserver (well actually it's just a pop proxy so it can also sit between your popfetch and the popserver - which is where I have it). It handles multiple accounts transparently and there is no limit in this regard

Once installed you tell popfile what types of email you wish sort between, a simple setup would be 'spam' and 'ok', which is how I started out. I soon realised that it's can be much more powerful than that and now have it sorting between 'spam', 'ok', 'lists', 'redhat' and 'virus' which it is doing nicely. Once it has worked out the type of email then it adds [spam] or [ok] etc. to the subject line - ready for you email client to sort on the message (you can turn this off in fact and use 'X-Text-Classification' so the subject line is not altered).

Popfile works by you 'training' it, this means it doesn't work right out the box but in my experiance it's actually very quick and easy to get it to do something useful. You set up any number of different things for it to differentiate between (it calls these 'buckets') when it receives an email the first time you tell it which bucket (e.g. 'spam') is and it then uses 'Naïve Bayes' to determine whether new emails are of similar type. Sometimes it will make a wrong decision but you simply goto it's config screen where it keeps a history of recent emails (configurable) and tell it to reclassify that particular email, the more you teach it the 'smarter' or more 'precise' it will become.

All config and 'training' of popfile is straightforward via your web browser to popfiles local http server (which by default is only connectable from the PC running popfile). You can also see statistics for how much mail of each type you have got and add remove buckets / sorting types. The screens are very well layed out and easy to follow. Screenshot of one of the screens is here

Popfile works on any OS that runs Perl (so nearly any OS!) I have it running under windows, when you download popfile for windows you can choose to put Perl on seperatly or use a popfile installer which adds Perl at the same time.

Popfile faq is here

I am very impressed with the way it works and the ease of use, the software is written totally in Perl, open source and free to use

Sil

edit, added the bit abt one account for mailwasher - apparently older versions did let you check more than one account and also connect to hotmail - these features are now only available from the 'paid for' version (!)

edit2, some additions to mailwasher text, added part abt manual intervention and improved description of 'race condition' scenario (hopefully!)

[silver]

This probably explains what popfile does better than I did

from old popfile faq

Meanwhile in the 21st century

Luckily, Bayes' 300 year old idea has a direct application to email sorting and text classification in general.

Imagine that you have three folders you'd like to sort email into: work, personal and spam (POPFile calls these folders 'buckets'). Setting up an email client to know how to sort the mail ranges from hard (in the case of work where you'd have to tell it about everyone in your company) and impossible (spammers keep changing their emails to evade filtering).

Bayes Theorem gives POPFile a way to calculate the probability that an email is work, personal or spam by calculating P(work|E), P(personal|E), and P(spam|E) where E is the new email and P(work|E) is the probability of email E being a 'work' email and so on. By picking the largest probability of the three POPFile can automatically pick the appropriate folder. POPFile calculates these probabilities by looking at the frequency with which words occur in each folder and applying Bayes Theorem.

Sil

PS, if anyone has other prefered mail sorting techniques or corrections to what I said please let us know

[Memfis]

This is actually a good point. I normally ban spam IP's from my server, But when they are relayed thru TS!'s addy I cant otherwise I'd be banning TS! & not them.

hmm which reminds of sommat else, (will post in IPCop thread).

~Mem

[Onslo]

Heh that looks kewl

I don't get too much spam as BTO filter both the virus and spam emails for me. However it's good to know should that system ever not be in place elsewhere

'Slo

[silver]

yeah - but it's not just for spam!

you can sort between any types of email - it's way easier and more accurate to use popfile to sort emails than using the 'from' address etc,.

it is

Sil

[bouncingjohn]

Originally posted by silver
The free version of mailwasher only supports one email account.. not really what I need at all.

That can't be right, I've got the free version and it's set up (and working) to check 3 accounts.

[silver]

You could be right bouncingjohn, I didn't actually install mailwasher, all information I have is after reading their faq (which could be wrong / misleading?)
,. it says here

Q. How many accounts may I have?
A. This version of Mailwasher only supports one account. If you wish to have multiple accounts, then upgrade to MailWasher Pro at http://www.firetrust.com.

Sil

[Ian]

The mail client that comes with Mozilla 1.3 has a version of bayesian spam filtering, works in a similar way, you teach it what is spam/not spam, takes a while to learn but its getting there

http://www.mozilla.org

A great server-side spam detector is spamassassin, this rates emails according to various criteria (eg, junk characters in body, lots of capitals, key words, etc,etc ) and gives each email a score, the higher the score the more likely it is to be spam.

It then adds a header to the email with the scoring and its up to you to decide what to do with it after that. (although OE and outlook dont support filtering on headers, many other email clients do) or possibly adds something to the subject ..

it can also use open relay blacklists, and Razor, a collaborative spam blacklisting service

http://spamassassin.org/

plus if you use qmail , qmail-scanner (scans emails for viruses) supports it

[silver]

I didn't know mozilla had that - sounds good

http://spamassassin.org/doc/Mail_Spa...sin_Bayes.html - sounds like it has bayes stuff in it tho not looked into it deeply. I looked into the http://razor.sourceforge.net thingy before and it sounds like a good idea but I guess it's a personal thing in that I don't want to rely on (or even use) external relay lists or md5 hash type stuff, not sure why really,.,. I guess that's why I like the bayes approach so much.

found a nice link which gives another outline of the difference between a bayes approach and rigid ruleset The Hindu : POPFile: An `organic' spam filter

Sil

[Ian]

noticed this on SA page:

http://saproxy.bloomba.com/

SAproxy features include:

Easy to use, universal: SAproxy is an easy-to-use anti-spam solution based on the open source SpamAssassinTM technology. If you use a POP3 mail client, you can use it to automatically detect incoming spam, mark it as spam, and then file it however you wish.

Works on most computers: SAproxy is designed to allow a Windows POP3 mail client to use SpamAssassinTM to filter your mail. Other versions of SAproxy are available from the Sourceforge page. Using the source code, SAproxy can be compiled for virtually any operating system.

[Tim]

Funny you should mention spam, there as been a massive increase into my work email box.

My collegues find it quite funny since I have had about 30 emails all exactly the same offering me enlargment services

I need a way to filter these and the Popfile sounds ideal. I will have to give it a go when I get chance.

Anyone tried Surfcontrol?? out of curiousity really.

[gem]

I have been using Mailwasher (free version) for nearly two months now and it checks all 13 of my email a/c's every 10 minutes. I have found it VERY usefull. It only loads the text so, in that sense, it does not load all of the email - you can pre-read it and decide whether to delete/bounce whatever. When I'm happy with whatever is left I click on 'Process mail' and it loads up (if it's not already loaded) Outlook (NOT express) and I hit F9 to download email. I used to have Outlook pick up email every 10 minutes but found that junk email as well as spam was getting to be annoying, hence Mailwasher.

[silver]

ok - will edit the bit abt mailwasher only doing one account

Sil

[silver]

gem / anyone - any idea how much of an email mailwasher downloads to check the content?

also, is there a race condition in the case where you have mailwasher running and your pop client running at the same time - so that your pop client can see mail on the server which mailwasher has not got round to dealing with?

actually, does it envolve a manual step in mailwasher to make the final determination of whether an email is spam and also a manual step to decide to delete / bounce the email?

I guess in some ways that mailwasher doing a partial download of each email to check the content is better than using a proxy which would generally see the whole thing b4 it decides what it is,, this is esp the case if you had lots of large spam emails or virus's which would be downloaded over dialup.,. tho the counter to this might be that getting the full thing allows more accurate determination (am on adsl so downloading email isn't a big issue!).

Sil

[gem]

Not quite sure what you mean, but here's a go.
Mailwasher loads the heading (sender, sent to, subject) but you hit the space bar to load the text of an email. When read, you can mark it for deletion, bounce etc. You can also blacklist the sender and/or the domain for future use.
When MW loads an email that has been blacklisted etc it marks each email accordingly but you can still preview it or even unmark it - so you have not unwittingly deleted that expected email in error. Only when the list is complete do you hit 'Process mail' does it take whatever action was clicked for each email.
Do come back if I missed something or not explained it well enough.
On the other hand, why not try it, it's free, it's only small and you can configure it to load up on statup or not, your choice. I find it very flexible.

[silver]

thanks gem

ok - the part abt race condition, say you tell MW to check popserver every 10 mins - but you set your email client (outlook / whatever) to check every 5 mins,.,. I would guess that your email client will see stuff b4 MW has had it's chance to deal with it?

also - from your example there does appear to be manual intervention required to get MW to "do it's thing" - which is ok I guess but then I could do it nearly as quick with popcorn (well ok not quite!)

I don't think I will install MW as I am more than happy with popfile (at least for now!) but I wanted to make sure my understanding of MW was correct (thanks for the corrections! )

Sil

[gem]

Race condition - I see what you mean. Yes, your email client would (as you say) stuff b4 MW has had it's chance. That's why I manually download email on Outlook AFTER previewing thro' MW.
MW requires user intervention to at least agree to it's suggestions if only by clicking on process email. The only 'automatic' part is it's 'learning' of the email address/domain to be accepted (friend) or deleted/bounced (blacklisted).

[silver]

yeah - that's reasonable, popfile does need some intervention (well it does when it makes a mistake abt what is what!).. don't get me wrong I think mailwasher is great for what people want it for, it is certainly a heck of a lot better than having nothing and it's free - can't really complain

but ... I like popfile, just a personal thing

Sil

[Scoobs]

im like gem and use MW pro i find its easy and works well you dont have to have outlook checking the mail as mw will do it at set times every x mins. you can preview all mail etc delete some and bounce and when you are ready click the button and it will open outlook and dl only the mail you want.

gem are you running version 3

scoobs

[gem]

I'm using the free version 2.0.16 but I've just downloaded 2.0.40.
I just re-read the blub. Version 2.0.40 is limited to one (1) email a/c! and no hotmail! I'll stick with 2.0.16 for the moment!
Thanks scoobs.

[gem]

Sil, please see my post above - the latest free version 2.0.40 IS limited to one a/c, sorry (mine is not!).

[silver]

ahh - that explains the confusion!

will go re-correct the correction

Sil

PS,. I do hope it's not the shape of things to come w.r.t. MW - they appear to be removing functionality from the free version in order to get peeps to pay for the 'pro' version - that's not really very customer friendly!

[bouncingjohn]

I'm on 2.0.16 as well, sorry if I misled you.

To expand a little on Gem's remarks: I believe (but don't take this as gospel) that MW parses each mail on your server looking for words and phrases that it knows are spam or spamlike - it will then show you a list of (all) the subjects/senders and you can choose what to do with them.

For example, I buy a lot of DVDs from Play.com and get sent their newsletter occassionally which MW detects as "possible spam", now I could leave MW's choice of bounce/blacklist but there's a (slight) chance that there's something interesting in there so - I add it to the "whitelist" and any future mails from that address are checked as "okay".

On the other hand (and one of the main reasons I got MW in the first place) I get a lot of mails from AShampoo (downloaded one of their players once and can't remember the password I used) from at least 3 different addresses. These I have set to be bounced/blacklisted. However, they will still appear in the list of mails waiting on the server so I could, if I wanted, download them.

Eventually, you should have a custom list of okay/not okay addresses and can be "reasonably" sure that MW has detected the mails you do and don't want to see.

*edit* oops, actually I'm on v2.0.21 beta.

[silver]

Thanks bouncingjohn

yes - I guess it could be said that setting up in the 'whitelist' is similar to 'training' in popfile, tho popfile handles this case in a more abstract way (tho in fact you can set it to do certain things based on sender / subject if you want).

I do still see an advantage to popfile in that there seems to be less intervention / manual steps required in the general course of things?

As an example when an email comes into popfile it's classified into one of the buckets (e.g. 'spam') that then sets the subject line to start with '[spam]', my email client it knows to put everything with a subject starting '[spam]' into the spam folder. Then whenever I am bored (once a day, once a week, whenever) I can go look through the spam folder in my email client and move it to trash, in the event I see an email that is wrongly classified I open up the gui on popfile and tell it that email wasn't spam. I am not bound to do this checking in the course of receiving emails, it's whenever I get round to it.

I quite like the sound of the bounce thingy but does it really stop them sending email? It might in some cases but a lot of spam is via open relay and the spammer will never get the bounce back, tho perhaps it might persuade the person who owns the open relay to do something abt it?!

Sil

[gem]

I have to say that while the 'bounce' does indeed bounce, as you say Sil, it never (or rarely) bounces back on the spammer. With MW you get a message to say that it could not bounce because .... so in that sense it means more user intervention in that you at least have to click on the response to acknowledge that it's failed to actually bounce (if you can understand that!).

[silver]

yeah,.,. well - bounce sounds handy but in practical terms I don't know how useful it is.. I guess it gives the user some feeling of 'fighting back' at least ?

If only the ppl who ran mailservers would go fix them to stop open relaying there would be far less spam or at least the spam there was may be easier to trace.

Sil

PS, if anyone gets round to trying popfile lemme know so then at least I am not the only one here running it!

[fabienne]

the .net CD has a full version of Ashampoo Mail Virus Blocker, has anyone tried it?

[gem]

Thanks fabienne.

[fabienne]

I've installed popfile and find it quite impressive, really. Within 5 attempts with a dozen mails it knows what to do, thx Silv!

Just a wee niggle: my Outlook E opens in folder Inbox, and i would like to keep this setting as it saves one click. <edit own nonsense>
However, when setting the rules, I can't choose "move to folder Inbox" option, as it greys out and will put the Ok mail in a new folder. Any idea on how i could get OE to open in the new folder, then?

[silver]

ahh, yeah I had the same thing...

on mine I have turned off 'subject modification' for the 'ok' stuff (it's in the bucket screen on popfile) - and within my mail client I simply let it stay in the inbox - i.e. there is no rule within my mail client to catch good stuff - just a rule to catch the nasties (and move those to 'spam')

Sil

PS, it's pretty spooky how quick it does something useful huh, tho it will still make mistakes esp at the start!