sharing internet connection load balancing between ISPs

silver · #1 03-September-2003, 10:21

Just because..

I am using ipcop's squid proxys (I have edited their conf's a little to stop squid actually caching and also to make them supply icp data) and I connect to another 'internal' squid proxy set to round-robin betwen the 2 ipcop proxies

works for http and ftp - note squid only supports ftp over http - so in flashget you have to change any 'ftp' urls to 'http',. i.e. replace ftp://ftpsite/url with http://ftpsite/url

basically it should be possible to make a failure of one ISP invisible to the user (if you setup squid right!) and generally you will get round robined between the ISP connections (so if you use something like flashget to give you 5 connects you will get a 3/2 split)

Sil

edit - by the way the 2 connections I have are adsl 512/256 and cable 150/64

edit - I have no idea what I was smoking when I thought switching ftp to http could possibly work,. needless to say it cannot and does not work...

Onslo · #2 03-September-2003, 12:12

That's pretty kewl

If I ever get another ADSL line then I might be asking for load balancing tips

'Slo

silver · #3 03-September-2003, 16:40

well - it's not 100% yet - pages stop loading properly when I turn off one of the PCs - but thats jus cos I haven't quite setup squid right

It took ages to try and make flashget proxy via the squid until I changed the url to 'http' and set the http proxy and then it jus worked

Sil

Tim · #4 03-September-2003, 18:32

That looks great.

I want to play lol.

Is it possible ot load balance everything? not just FTP/http??

I know you can get devices to, but is it not possible to do it with a handful of computers??

silver · #5 03-September-2003, 19:46

there is!

but requires too much magic for my time contraints,,. basically (from my limited understanding!) you need a magic NAT that can do load balancing, also it would need to be able to fake the source IP if you wanted it to look like it came from the same IP (some things like p2p or games might not like to see other IPs)..,.

really a lot depends on exactly what situation you are trying to cope with - some links on load balancing

http://lartc.org/howto/

http://perl.apache.org/download/thir...ncing_Projects

if you want to balance / high availibility an incoming connection (e.g. webserver) then there are some free round robin type dns solutions around

I am happy with doing the squid proxy,. it is relativly straightforward and simple enough for me to understand how it works !

Sil

PS,. the apache link shouldn't have a space in it - dunno why it has!

Ian · #6 04-September-2003, 09:47

Heres a good page (bar the green text on a black background) about various methods of using 2 or more links.

http://leaf-project.org/doc/howto/LR...ing-HOWTO.html

silver · #7 04-September-2003, 10:15

thanks ian - looks useful

Sil

silver · #8 09-September-2003, 15:41

ok - just to add a little more detail onto how the setup works

Code:

    ISP1          ISP2
     |             |
+--------+    +--------+
| IPcop1 |    | IPcop2 |
| Squid1 |    | Squid2 |
+--------+    +--------+
     |            |
     +------------+--------+--------------+
                           |              |
                      +--------+     +--------+
                      |        |     |        |+
                      | Squid3 |     |Clients.||
                      |        |     |        ||
                      +--------+     +--------+|
                                       +-------+

was a little bored

right - basically I have tweaked the ipcop's squid proxies so they don't cache anything (no need really) and turned on icp (not sure this is 100% necessary though!).. the basic magic is done in 'Squid3' - which have a normal default gateway setup to one of the ipcop boxes - but the squid proxy settings are set to round robin..

Code:

cache_peer 172.25.90.222 parent 800 801 no-query no-digest round-robin
cache_peer 172.25.90.111 parent 800 801 no-query no-digest round-robin

the 801 being the port that I've set ipcop to use for icp_port on each of the ipcops,. the 172.25.90.xxx are the 2 ipcop boxes green or internal ips

to make any squid changes on the ipcop boxes you need to edit /home/httpd/cgi-bin/proxy.cgi file to include things like ipc_port and turn off caching - then restart them to make the changes take effect.

then set the clients up to use the Squid3 proxy (or use a pac file loaded from an internal webserver whatever)..

it works reasonably well and does seem to cope with one of the ISPs going offline now

Sil

silver · #9 09-September-2003, 15:46

here's the bits I changed in ipcop's proxy.cgi

Code:

        print FILE <<END
shutdown_lifetime 5 seconds
#silver - changed to a real value
#icp_port 0
icp_port 801

http_port $netsettings{'GREEN_ADDRESS'}:800

#silver - comment out
#acl QUERY urlpath_regex cgi-bin \\?
#no_cache deny QUERY

cache_effective_user squid
cache_effective_group squid

pid_filename /var/run/squid.pid

cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
log_mime_hdrs off
forwarded_for off

acl all src 0.0.0.0/0.0.0.0
#silver - stop caching all acl
no_cache deny all
acl localhost src 127.0.0.1/255.255.255.255
acl localnet src $netsettings{'GREEN_NETADDRESS'}/$netsettings{'GREEN_NETMASK'}

#silver - added block on 'via' and 'forwarded-for' - useragent is faked anyway
anonymize_headers deny Via
anonymize_headers deny X-Forwarded-For

END
;

Sil

rarr · #10 10-September-2003, 12:13

Just because I ca also...

2 512kbs adsl lines with load balancing

silver · #11 10-September-2003, 12:56

Sil

PS, should also link <thread>:Outgoing load balancing between two ADSL lines which has a couple of other ways of splitting the outbound traffic (e.g. netsplitter)

hmiddelhoek · #12 09-March-2004, 19:10

I need to kick this topic

About load balancing.
A few things aren't obvious to me. When using 2 internet connections, are you using 3 ipcop's then? 1 for each connection and one to balance the traffic between the two ipcop's/connections.

Is what rarr showed us the result of your method silver? Load balancing with squid? But you need to put proxy setting in the cliënts. And they can throw out these settings.

Are there (some months later) any new tricks to load balance between two connections?

Thanks in advance.
Hans Middelhoek

silver · #13 09-March-2004, 21:18

Hi Hans,

yes, I used 3 squids total, 2 squids on ipcop boxes and a 3rd squid box that round-robins between them (not ideal!)

some interesting info http://forums.techwatch.com.au/viewt...?t=3626&sid=22 - esp the part abt smoothwall - perhaps?

rarr's method uses netsplitter - but I don't know much abt that at all.. it's not proxy based but uses some routing magic to push traffic across the different connections (so far as I can tell).

You should be able to do something similar in the kernel now using ip route and multiple tables - I got 95% of the way there in <thread>:ADSL and ISDN as Backup - to make this load balance between the 2 connections you jus need to use a multi-hop when the gateway isn't already known (umm - it's some time since I played with it so I can't remember now,. I know it wasn't much harder - the links I gave in that thread should say how)

tell us a bit about the exact scenario you are trying to deal with ?

Sil

PS, welcome to TS!

hmiddelhoek · #14 10-March-2004, 22:57

It's about two ADSL connections, each in a different village. This connections go to another central village with a wlan. From there it's split. At that place there needs to be a router which load balances the traffic between the two wlan connections each to another ADSL connection.

Code:

          Internet
         /        \
village 1         village2
     |                |
 router             router
     |                |
wlan AP           wlan AP
      \               /
       \             /
        \           /
wlan AP village 3 (central)
                 |
             switch_______eth0-|-------------|
             |  | |_______eth1-|Load Balancer|
             |  |_________eth2-|-------------|
             |
     Other AP's

It's just a global view of the network. Without workstations drawn, but it's clear I think. One thing to tell, it isn't my own network, I'm just helping someone without knowledge of routing, ip structure and linux. He set up this network and we are improving it right know. Because of the 40+ users the 2 Mbit link (down) looks to little, so a 4 Mbit is ordered. When it is connected we need to load balance between the two connections.
When one wireless connection failes or a ADSL link is offline there always is another path to go

That's why the three network interface cards in the load balancer (linux). 2 nic's each his own gateway and 1 nic as the local gateway for the intranet. I think we're going to use a second switch for the second uplink. The AP's are splitten between the two switches and the switches connected to eachother. So we can build some redundancy. When a link failes everything goes well I think, but when a switch failes nog the whole network is down but only a part of it. Another switch is in stock to solve this.

I think I can do all this by using the manual: http://lartc.org/howto/lartc.rpdb.multiple-links.html
I got the link out a earlier post about this subject.

Hope you can advise me or react on my story. Maybe I'm thinking in the wrong direction.

Thanks for the welcome and your reaction.
Hans

silver · #15 08-August-2005, 08:21

this is similar to the squid method - but will balance tcp on a packet by packet basis http://www.inlab.de/balance.html or http://sourceforge.net/projects/balance/

Sil

kiaraliz · #16 15-September-2009, 05:14

ok - just to add a little more detail onto how the setup works
Code:
    ISP1          ISP2
     |             |
+--------+    +--------+
| IPcop1 |    | IPcop2 |
| Squid1 |    | Squid2 |
+--------+    +--------+
     |            |
     +------------+--------+--------------+
                           |              |
                      +--------+     +--------+
                      |        |     |        |+
                      | Squid3 |     |Clients.||
                      |        |     |        ||
                      +--------+     +--------+|
                                       +-------+
was a little bored

right - basically I have tweaked the ipcop's squid proxies so they don't cache anything (no need really) and turned on icp (not sure this is 100% necessary though!).. the basic magic is done in 'Squid3' - which have a normal default gateway setup to one of the ipcop boxes - but the squid proxy settings are set to round robin..
Code:
cache_peer 172.25.90.222 parent 800 801 no-query no-digest round-robin
cache_peer 172.25.90.111 parent 800 801 no-query no-digest round-robin
the 801 being the port that I've set ipcop to use for icp_port on each of the ipcops,. the 172.25.90.xxx are the 2 ipcop boxes green or internal ips

to make any squid changes on the ipcop boxes you need to edit /home/httpd/cgi-bin/proxy.cgi file to include things like ipc_port and turn off caching - then restart them to make the changes take effect.

then set the clients up to use the Squid3 proxy (or use a pac file loaded from an internal webserver whatever)..

it works reasonably well and does seem to cope with one of the ISPs going offline now

Sil
Originally Posted by silver

Hi Sil.
Could you help me find a solution to our problem? Upon reading this topic I believe this will give as a hint on how we could accomplish our major task which is to speed up internet connection using two PPP connections.

I'm still a newbie and I am confused if IPCOPs and SQUID are all running on a single linux computer?

Could you help me (in detailed please ^^) on how could I configure IPCOPs and SQUID to be able to come up with the kind of topology you have designed above..thanks..i would really appreciate your help..relly urgent..

Regards,
Kiara

silver · #17 15-September-2009, 13:26

there's multiple ways to share connection between 2 ppp (dialup?) ISP connections, describe what is in place currently, I mean in terms of the current phyisical topology - 2 dialup lines into how many computers or routers?

kiaraliz · #18 22-September-2009, 16:01

there's multiple ways to share connection between 2 ppp (dialup?) ISP connections, describe what is in place currently, I mean in terms of the current phyisical topology - 2 dialup lines into how many computers or routers?
Originally Posted by silver

Hi Sil.

Thanks for the reply.

What I am trying to do is to use two usb 3G modems (treated ppp0 and ppp1 by ubuntu) into one pc to speed up the connection. I am only using one laptop. I have been trying to do load balancing because I was able to enable both usb modems however, the traffics/packets are routed to only one ppp, specifically ppp1 (the last modem connected). I have read from other forums that to be able to do this, the service provider should support multiple ppp or multilink. I tried using two usb modems from the same provider and another is I tried using two usb modems from different providers but still I could not use both modems.

Now, since I cannot find a solution to my problem I am planning to do what you have done which is to use 2 IPCOPs and a SQUID. However, I am still a newbie and I don't know where to start. Please help I just have to pove that I can speed up my connection using two usb modems, in any way possible.
I would really appreciate your help. Really in need.

Thanks again.

Regards,
Kiara

silver · #19 22-September-2009, 21:15

ah - well then I might suggest you check out http://www.pfsense.com/ - supports dual wans

the behaviour of the box picking the most recent ppp connection is for default routing,. it's got to pick one to shove outbound traffic and last one is as good as any

not sure if http://m0n0.ch/wall/ supports dual wans but pfsense is/was originally derived from that

when I came up with the slightly inefficient 3 squid solution I didn't know that much about how routing tables work (hey that was 6 years ago!) now I might come up with something slightly better,. or at least I'd just put all the squids onto one box and have them bind to specific network interface - which I think must be possible

silver · #20 22-September-2009, 21:18

btw - the term "connection load balancing" (as used by me at least) can mean all kinds of things,. describe what kind of traffic you need to speed up,. e.g. is it websurfing or uploading to ftp servers etc?

kiaraliz · #21 22-September-2009, 23:58

Hi Sil,

I would like to establish faster connection to the Internet, specifically about websurfing and uploading/downloading files to to and from the internet. I am using 2 Huawei E220 USB HSDPA modems to establish the connection. When I brought up both devices into the Linux computer, only one device was used to send and receive the packets. What I would like to do here is be able to use both devices to load balance the traffic.Another problem I encountered in using the devices is that the IP address is dynamically assigned, and the default gateway is not known.

Could you suggest any solution to this?

Thanks

silver · #22 23-September-2009, 00:18

see post above,. I'm reasonably sure pfsense could do what you need if it supports the modem (which it should if you have had them working under linux)

there's a lot more info on this than there was all that time ago,. try google with http://www.google.com/search?q=dual+wan+linux - is stuff there to deal with the routing tables, though personally I would check out pfsense

btw not tried pfsense as I make do w/ one i'net connection these days,. if I was going for dual wan that is where I would start

kiaraliz · #23 23-September-2009, 01:35

Hi Sil.

We tried installing pfsense just now but it does not recognize ppp connections. Hmmmm, do you have any other suggestion? Sorry for asking too much. Thanks

kiaraliz · #24 23-September-2009, 02:36

BTW, I have also heard about ZeroShell that could handle ppp connections. However, I could not install it. I would search for other solutions. Thanks really.

kiaraliz · #25 23-September-2009, 03:38

It's me again.

I have read that rarr used netsplitter. I have downloaded it but I could not find steps on how to install and make it work. I am willing to try different solutions because I really have to prove that internet connection will be improved if i use two usb 3G modems.

I would just like ask for your help on how to use netsplitter.
Thanks again.

silver · #26 23-September-2009, 09:08

Hi Sil.

We tried installing pfsense just now but it does not recognize ppp connections. Hmmmm, do you have any other suggestion? Sorry for asking too much. Thanks
Originally Posted by kiaraliz

it should do,. your modem is on the supported hardware list http://www.freebsd.org/releases/7.0R/hardware.html - so it must work with ppp

also if you really want just a proof of concept then get 2 ipcop boxes - put the ppp connection direct to that then put pfsense behind them,. so it's ethernet to pfsense (though direct using ppp should work)

silver · #27 23-September-2009, 09:18

It's me again.

I have read that rarr used netsplitter. I have downloaded it but I could not find steps on how to install and make it work. I am willing to try different solutions because I really have to prove that internet connection will be improved if i use two usb 3G modems.

I would just like ask for your help on how to use netsplitter.
Thanks again.
Originally Posted by kiaraliz

I think netsplitter is pretty well no longer suported / updated (from what I could see and also I have not used it!) - found http://cwareco.com/eqlplus.html which might work ok

kiaraliz · #28 24-September-2009, 12:16

Hi Sil.

Thanks again for your suggestion. Anyway, we could not successfully install eqlplus. There were errors when we did the patches. Hmmm, we would really want to find the best solution but we are losing hope. We are planning to change our topic. Thanks anyway for your help. I really appreciate it.

Memfis · #29 19-October-2009, 17:35

http://ask.slashdot.org/story/09/10/...SP-Connections
also search the forums for the following keywords :
mlppp
bonded
bonding

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode