linux Q: good advert cookie blocking s/w

[silver]

I would like to make my gateway PC linux (have been wanting to do it for ages!) an I think now ADSL on USB is more or less working (?) so it should be possible..

pref want to use an http proxy on the linux box that can filter adverts by url string matching an also cookie blocking using some sort of white list - logging on whats blocked etc is needed

any ideas?

thanks

Sil

PS, as extra requirements the ability to block on image size (tho not sure how effective this is really) an it would be v nice if it could do page editing on the fly to remove flash an fiddle out nasty jscript,,, referrer blocking/faking would be nice 2..

[Ian]

Err, I had loads of suggestions, until I saw your list of demands

I use Squid and Sleezeball to block banners + sites. (dead, dead easy to install/configure)

Theres also Squidguard similar principle to sleezeball, more involved to setup (and probably more suited to a business environment) but more options.

I also have a iptables (firewall) blacklist of subnets of the most companies (IMHO ) which prevents *any* access in or out on any ports.

(For this I use Shorewall *much* more straightforward to set up than it first seems. )

One program that promises all you want is FilterProxy but having waited *ages* for a version that didn`t rely on an obsolete perl module, it turned out much too slow for my liking (noticable delay even on a dial up) although it should be ok on a reasonably pokey pc (ie not a pentium 166)

http://www.webwasher.com/ do a free linux version with a nice web based interface, although its limited to 2 ips connecting to it.

And finally this is one I saw recently, its based on junkbuster with some other bits bolted on Privoxy

[silver]

kewl

the only one I knew offhand was junkbuster,.,

I know muffin would work - an that can do all that - web filtering unfort they stopped work on it by the looks of things

pity proxomitron doesn't have a linux version..

I guess I could fiddle with the src of junkbuster (well mebbe)

Sil

[silver]

humm - Perl based proxy that does html filtering (on the fly page editing) + cookie blocking.. http://www.lne.com/ericm/cookie_jar/

Sil

PS, should be easy to hack..

[silver]

other links

http://www.flourish.org/adremove/

an

http://www.junkbusters.com/links.html

has some really useful sounding stuff

will check some out when I get a chance

Sil

[silver]

should include a link to the ODP

http://dmoz.org/Computers/Software/I...ng/Ad_Filters/

Sil

[silver]

not sure this is what I want - but just linking

http://dansguardian.org/

DansGuardian is a web content filter which currently runs on Linux, FreeBSD, OpenBSD, NetBSD, Mac OS X, and Solaris. It filters the actual content of pages based on many methods including phrase matching, PICS filtering and URL filtering. It does not purely filter based on a banned list of sites like lesser totally commercial filters.
DansGuardian is designed to be completely flexible and allows you to tailor the filtering to your exact needs. It can be as draconian or as unobstructive as you want. The default settings are geared towards what a primay school might want but DansGuardian puts you in control of what you want to block.

I didn't get round to installing a proxy filter but I do need to do it at some point, one that uses a cookie jar based on some type of user login / source IP address (so that different PCs/users cookies get stored / used seperatly) would be handy.

Sil

[silver]

ano link http://www.math.grin.edu/~lindseyd/paper.html

Sil

[Memfis]

Wouldn't IPCop do all of this?

supports USB ADSL, has proxy, can filter, can run dans gardian, and can cache.

and I'm sure you already run IPCop?

have I lost the plot here???????

~Mem

[silver]

yeah - it could run a web proxy, in fact it has one already (squid), personally I don't think a firewall / router should run a web proxy (I have turned it off on mine - which is the default). The question isn't where to run it tho (firewall or on another PC) the bit I need is a decent advert / cookie filtering solution, squid alone doesn't provide it.

Will re-state requirements (for my benefit!):

should keep cookies local to the proxy (like junkbuster does with cookie jars) preferably with the ability to have 'user profiles' so that each user has a different cookie jar, cookie jar should have a white list (based on 'user profile') and some way of accepting cookies temporarily for later deletion (possibly)

should block adverts / images / flash based on the url text, e.g. if it contains '.ad.' in the url should be blocked - block certain domains

should possibly be able to block images based on size (not that necessary)

referer faking / blocking should be possible - not strictly necessary

I think that pretty much covers it (at the moment)

Sil

PS, a new (old) PC has come into my possession so I now have the box to do this on (once the new box is online!)

[Memfis]

hmmm you keep falling over these "old" PC's.

I still cant run IPCop as no spare PC.

when I did run it (tho for a very short time) I uploaded a new hosts file combined from multiple free ones on the net with all ad servers etc pointing to 0.0.0.0

That cut out 99% of the ads that I saw.

no longer running IPCop (as above) however still maintain a local hosts file on my workstation to prevent ads.

~Mem

[silver]

well - I thought it was slightly out of date - but junkbuster hasn't been updated since 1998!

it's gpl'd and there's a few branches with new names and new stuff

am looking at http://www.privoxy.org which atm I am thinking of putting in front of a squid proxy on my gentoo linux box

lots of good resources listed at http://www.flourish.org/adremove/index.html

Sil

[silver]

I am not sure I understand all the ins and outs of either squid or privoxy (not that I normally let that stop me! ) but it appears to be working,. the privoxy site says to put it together like

client browser > squid > privoxy > internet

which I have,. an the chain all appears to work

there is some *apparent* slowdown if you have privoxy html filters on (by default it looks at the html and edits it on the fly - which is neat!) this means it won't pass the html to the browser until it has seen the whole page (which is why there's an apparent slowdown),. you can exclude sites from this html filtering if there's sites that you don't need to do this for (it will still block adverts on those sites by their urls etc)

putting squid into the chain also removes latency and all in all I am impressed (squid didn't start right away - you have to edit the config an do various changes to chain it to privoxy)

using squid and privoxy meets most of my requirments, the one it doesn't meet I am not sure anyone has done it

should keep cookies local to the proxy (like junkbuster does with cookie jars) preferably with the ability to have 'user profiles' so that each user has a different cookie jar, cookie jar should have a white list (based on 'user profile') and some way of accepting cookies temporarily for later deletion (possibly)

this was based on my mis-understanding of the junkbuster cookie jar - I assumed the jar was used to feed the websites with cookies - but it isn't - it's really just a trashcan for cookies that were blocked,.

I would still like this ability as I think pushing any cookies back at the browser is unnecessary when a proxy can do it with more control (esp if you can add rules - like delete cookies after 1 hour etc) privoxy does block cookies and you can have a cookie domain white list - so it's not too far off

Sil

PS,. privoxy is looking to add user profiles - so I think the ability to do cookies for users is perhaps coming

[Ian]

how did you configure squid to forward requests through privoxy ? and does it still cache stuff ? (the last time i set squid up to forward to another proxy it didnt do any caching..)

[silver]

yes - it's not simple at all is it!

jus follow the bit at

http://www.squid-cache.org/Doc/FAQ/FAQ-4.html#ss4.9

4.9 How do I configure Squid forward all requests to another proxy?
Note: The information here is current for version 2.2.

First, you need to give Squid a parent cache. Second, you need to tell Squid it can not connect directly to origin servers. This is done with three configuration file lines:

cache_peer parentcache.foo.com parent 3128 0 no-query default
acl all src 0.0.0.0/0.0.0.0
never_direct allow all

Note, with this configuration, if the parent cache fails or becomes unreachable, then every request will result in an error message.

I am not familar with squid at all,. but it is def caching some things some times,. if I click on a link that takes me to the page I'm on I hardly get any misses

e.g.

1060869627.879   2357 172.25.199.101 TCP_MISS/200 46246 GET http://www.the-scream.co.uk/forums/index.php? - DEFAULT_PARENT/172.25.199.103 text/html

is the entry from '/var/log/squid/access.log' for when I load the forum,. by pressing 'forum'

privoxy is a much simpler proposition to set up,. it's all pretty much browser based config (except initial listening IP + port I think) the gui isn't exactly intuitive - but it is functional

privoxy bahaves a lot like muffin - editing html on the fly to remove dodgy jscript etc - v nice

Sil

[silver]

opps - I added a tweak to stop it asking for a cache digest

cache_peer 172.25.199.103 parent 8118 0 no-query no-digest default

Sil

[Ian]

aah, but do you get any hits and does your cache directory grow in size ?

[silver]

well - the cache is growing

du -s /var/cache/squid
18368   /var/cache/squid

I think you are right tho - it isn't caching - since I don't see any tcp_hit in the logs,. am trying to fix it

http://www.squid-cache.org/Doc/FAQ/FAQ-6.html

has info on what the tcp_miss etc mean,.

Sil

[silver]

humm - it is confusing,.

how do you check if it's caching correctly (i.e. can you give me a series of pages to load and what to look at to know if it's caching ok) ?

thanks

Sil

[Ian]

You should be seeing lots of

TCP_REFRESH_HIT

in the access logs if it is caching stuff

You may have to put privoxy before squid, or use something like squidguard or sleezeball if you just want to block sites/ads - which work as squid redirectors

[silver]

I think I fixed it (not totally sure yet!)

in the

cache_peer 172.25.199.103 parent 8118 0 no-query no-digest default

it should be

cache_peer 172.25.199.103 parent 8118 7 no-query no-digest default

i.e. port 7 - it says that makes it use the udp echo port,. but I wasn't running the echo-udp service (had to put xinetd on!) an I also started the tcp echo service (not sure that it makes any difference!)

anyway - looking at the log,. it does appear to be caching,.

I am seeing mostly

TCP_HIT/200
TCP_MEM_HIT/200
TCP_IMS_HIT/304

which shows they are being cached ?

,.

the way I'm testing - goto a new site (so squid hasn't seen it) - load the page,. (lots of cache miss right - as expected) ,,,

then in IE I delete temp files (an all offline content) so IE doesn't have the page cached,.

then I go back to the page using a link (to avoid pressing 'refresh' on the browser)

an I see those 3 types of HIT's - so it's fixed?

Sil

[silver]

cache is def going up

du -s /var/cache/squid
24176   /var/cache/squid

Sil

[silver]

am now seeing

TCP_REFRESH_HIT
The requested object was cached but STALE. The IMS query for the object resulted in "304 not modified".

which I guess shows it's working?

Ian, do you use any kind of squid stats package or a browser based i/f - if so which as I am wondering if I should install one?

Sil

[Ian]

no, the access.log gets huge, very quickly, i`ve only got a few 100MB free diskspace, so i`ve turned it off

[silver]

ah ok,. I guess as long as it's working I don't need to check it at all really

I might turn off logging or make sure it's on logrotate or whatever,.

I have put in a gig of squid disk cache - which should be plenty

Thanks

Sil

PS,. am slowly getting my head round privoxy an beginning to think it does indeed rock,. the giu could do with being slighly better organised but then again most things don't have gui's and you can do it all by editing the conf files direct so I shouldn't complain,. all in all though it has a lot of neat features

[Kletzx]

Hi
Havnt U tried Safesquid
It ll surely satisfy all ur needs
Its an excellent content filtering proxy with all ur needs
Very good content filtering proprties based on url blacklisting,dns blocking,key word filtering and much more
And Yaaah,it has image filtering also
It can be made to work in conjuction in squid if u need to do so
Ge more info frm their site,also a trial ver ca be downloaded frm there
www.safesquid.com
They are also maintaining a good forum there
Try it
Cheers

[doofus]

Will re-state requirements (for my benefit!):

should keep cookies local to the proxy (like junkbuster does with cookie jars) preferably with the ability to have 'user profiles' so that each user has a different cookie jar, cookie jar should have a white list (based on 'user profile') and some way of accepting cookies temporarily for later deletion (possibly)

should block adverts / images / flash based on the url text, e.g. if it contains '.ad.' in the url should be blocked - block certain domains

should possibly be able to block images based on size (not that necessary)

referer faking / blocking should be possible - not strictly necessary

I think that pretty much covers it (at the moment)

Sil

PS, a new (old) PC has come into my possession so I now have the box to do this on (once the new box is online!)

Originally Posted by silver

I'd strongly recommend using this "old PC" to load up a firewall machine running with OpenBSD, using pf and NAT enabled. If you're okay with Linux, the shift over won't be too bad at all. You'll be able to tune those pf rules so tight, and using a relay with sendmail and the rules before you even get to your preferred email client you'll probably be able to sort out 99% of the chaff from the wheat!! There's plenty of docs and OBSD can't be beaten for security. As they state on their web-site (http://www.openbsd.org): only 1 remote hole in over 8 years, just with the default installation!! Not bad going, because it was built from the ground up with proactive security in mind. You can, like any OSS, get it for free (though they don't do iso's - don't get fooled into claims: those are bogus!!), but if you do get it, then consider supporting their work by buying the next installation (six monthly developmental cycles).
Question is though whether you want to scale the learning curve??

Cheers

[silver]

thanks doofus,

Good advice

I do have a firewall box setup already, ipcop <thread>:Make Your Own Router - secrets revealed which I like for it's ease of setup / use,. and also have several debian boxes behind it in a dmz so I know a little unix-like

Have briefly looked at openbsd and I agree it's def one of the more secure setups (it was that or freebsd, not sure now!) debian tho seemed easier to get setup quickly,. no doubt I will try one of the bsd's at some point!

Sil

PS, going back to thread topic,. I now don't bother too much about cookies or blocking adverts,. IE6 (eww!) has ok cookie handling and google toolbar stops most popups,. guess I don't go to that many sites these days or enough to worry abt advert blocking

PPS, welcome to TS!