Load Balancing two ISPs

Today I finally managed to use my two ISPs together on my desktop, combining both bandwidths into one big (almost 3Mbit!) pipe.

My setup:

1. ADSL modem 1Mb down, 320Kb up
2. GSM modem 2Mb down, 512Kb up

Configuring both at the same time is simple, but then we have two default gateways and our packets always go out through the first one found (or with the lower cost as defined in the ‘metric’ parameter.)

So how can we divide our packets over both links?

Googling around I found several suggestions to use iptables.
The general idea is:

• use -m statistic in a chain to choose packets to use on of two links (either the ‘nth’-method or the ‘average’-method
• set a mark on the packet
• use an ‘ip rule’ to select a routing table for mark 1, mark 2, etc.

That sounded like a perfect solution. This way I could really balance my two links like 40%/60% or whatever.

But it didn’t work…

My desktop is not a router, so I have to treat the packets in the OUTPUT chain, where routing has already taken place. The above-described method works on Linux routers treating the PREROUTING chain in iptables, where we can mark a package before routing.

So I studied IP ROUTE and IP RULES a bit more, browsing through the fantastic Linux Advanced Routing & Traffic Control site.

I discovered that we can use ‘nexthop’ to ‘hop’ between several routes.
After experimenting a bit I wrote the following script::

#!/bin/bash
#
# bal_local        Load-balance internet connection over two local links
#
# Version:         1.0.0 - Fri, Sep 26, 2008
#
# Author:          Niels Horn
#

# Set devices:
DEV1=${1-eth0}    # default eth0
DEV2=${2-ppp0}    # default ppp0

# Get IP addresses of our devices:
ip1=`ifconfig $DEV1 | grep inet | awk '{ print $2 }' | awk -F: '{ print $2 }'`
ip2=`ifconfig $DEV2 | grep inet | awk '{ print $2 }' | awk -F: '{ print $2 }'`

# Get default gateway for our devices:
gw1=`route -n | grep $DEV1 | grep '^0.0.0.0' | awk '{ print $2 }'`
gw2=`route -n | grep $DEV2 | grep '^0.0.0.0' | awk '{ print $2 }'`

echo "$DEV1: IP=$ip1 GW=$gw1"
echo "$DEV2: IP=$ip2 GW=$gw2"

### Definition of routes ###

# Check if tables exists, if not -> create them:
if [ -z "`cat /etc/iproute2/rt_tables | grep '^251'`" ] ; then
   echo "251    rt_dev1" >> /etc/iproute2/rt_tables
fi
if [ -z "`cat /etc/iproute2/rt_tables | grep '^252'`" ] ; then
   echo "252    rt_dev2" >> /etc/iproute2/rt_tables
fi

# Define routing tables:
ip route add default via $gw1 table rt_dev1
ip route add default via $gw2 table rt_dev2

# Create rules:
ip rule add from $ip1 table rt_dev1
ip rule add from $ip2 table rt_dev2

# If we already have a 'nexthop' route, delete it:
if [ ! -z "`ip route show table main | grep 'nexthop'`" ] ; then
   ip route del default scope global
fi

# Balance links based on routes:
ip route add default scope global nexthop via $gw1 dev $DEV1 weight 1 \
   nexthop via $gw2 dev $DEV2 weight 1

# Flush cache table:
ip route flush cache

# All done...

You can download the script here from my homepage.

This is not the perfect solution, as routes are cached, so once you connected to an external site, it will continue to use the linkt hat was originally selected.
So an FTP download won’t benefit from this solution, but torrent downloads will, as they use several parallel connections.

I tested the result and managed to download using BitTorrent with the incredible speed of 250KBytes/sec:

Update: This script is now also part of an article on slackwiki.