This is an old revision of the document!
Table of Contents
Networking notes
ip route & multiple routing tables
For routing specific client traffic through specific interfaces
Desired Config: An openVPN client connection running from a Tomato/OpenWRT/DDWRT router, routing traffic for only some of our LAN clients through the VPN, while all others route through the default ISP connection.
Enter “ip route” (I've always just used the default “route” in the past). “Ip route” allows the creation of multiple routing tables and “rules” to direct the traffic to a specific table.
Details on everything are below, or cut to the chase.
openVPN client settings
My VPN provider sets a bunch of routes when making the connection, if you let it. If you don't want that you need to set the openVPN directive “route-noexec”. In Tomato, there's an option called “Create NAT on tunnel”, that is checked. Under Advanced settings, my openVPN client settings are like so:
persist-key persist-tun tls-client auth-user-pass /tmp/auth.txt comp-lzo verb 1 reneg-sec 0 route-noexec route-up /tmp/vpnrouteup.sh down /tmp/vpnroutedown.sh
The contents of /tmp/auth.txt are simply VPN provider username on the top line, password on the second line. The contents of the route-up and route-down scripts are at the bottom of this page.
For those interested, the “Create NAT on tunnel” option executes the /tmp/etc/openvpn/fw/client1-fw.sh script, which looks like:
#!/bin/sh iptables -I INPUT -i tun11 -j ACCEPT iptables -I FORWARD -i tun11 -j ACCEPT iptables -t nat -I POSTROUTING -s 192.168.1.0/255.255.255.0 -o tun11 -j MASQUERADE
ip route commands and new routing tables
From what I've seen there appear to be 3 default tables:
- main
- local
- default
This can be confirmed via ip rule list:
root@wifi1:/tmp/home/root# ip rule list 0: from all lookup local 32766: from all lookup main 32767: from all lookup default
Other tables are defined with integer names. To create a new table simply add a route to it with ip route add [route] table #tablename#:
ip route add 192.168.100.100 via 127.0.0.1 dev lo table 100
To see the table, ip route list table 100:
root@wifi1:/tmp/home/root# ip route list table 100 192.168.100.100 via 127.0.0.1 dev lo
To add a rule to affect traffic and direct it via routes in your new table 100, ip rule add [blah] table 100, and to see the rule in place, ip rule list:
root@wifi1:/tmp/home/root# ip rule add from 9.9.9.9 to 192.168.100.100 dev lo table 100 root@wifi1:/tmp/home/root# ip rule list 0: from all lookup local 32765: from 9.9.9.9 to 192.168.100.100 iif lo lookup 100 32766: from all lookup main 32767: from all lookup default
That rule (as far as I can tell) tells the kernel that when traffic arrives on interface lo from ip 9.9.9.9 with destination address 192.168.100.100, then the pertinent routes are found in routing table 100. If you want to direct traffic both ways to and from specific ip's, then you must do so explicitly:
root@wifi1:/tmp/home/root# ip rule add to 9.9.9.9 from 192.168.100.100 dev lo table 100 root@wifi1:/tmp/home/root# ip rule list 0: from all lookup local 32764: from 192.168.100.100 to 9.9.9.9 iif lo lookup 100 32765: from 9.9.9.9 to 192.168.100.100 iif lo lookup 100 32766: from all lookup main 32767: from all lookup default
If you make a mistake and want to delete those rules, ip rule delete pref #rule#:
root@wifi1:/tmp/home/root# ip rule delete pref 32764 root@wifi1:/tmp/home/root# ip rule delete pref 32765 root@wifi1:/tmp/home/root# ip rule list 0: from all lookup local 32766: from all lookup main 32767: from all lookup default
routes set by VPN provider
In my case, the routes that get set by my provider look like so:
root@wifi1:/tmp/home/root# ip route show |grep tun11 10.173.1.5 dev tun11 proto kernel scope link src 10.173.1.6 10.173.1.1 via 10.173.1.5 dev tun11 0.0.0.0/1 via 10.173.1.5 dev tun11 128.0.0.0/1 via 10.173.1.5 dev tun11
My TUN ifconfig in that case looks like:
root@wifi1:/tmp/home/root# ifconfig tun11 tun11 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:10.173.1.6 P-t-P:10.173.1.5 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:17428 errors:0 dropped:0 overruns:0 frame:0 TX packets:19874 errors:0 dropped:16 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:9120474 (8.6 MiB) TX bytes:3519467 (3.3 MiB)
We can see then that our tun11 IP is .6, our gateway (which is “via” in the table) is .5, and then some default routes are set.
directing the traffic for a specific IP
To make these routes work for only some of our LAN clients, we need to add them to a new routing table, and then specify rules to make use of them.
In my case, I'm going to route only traffic from 192.168.1.253 through the VPN.
Manually, that would go like this:
ip route add 10.173.1.5 dev tun11 proto kernel scope link src 10.173.1.6 ip route add 0.0.0.0/1 via 10.173.1.5 dev tun11 table 10 ip route add 128.0.0.0/1 via 10.173.1.5 dev tun11 table 10 ip route add 10.173.1.1 via 10.173.1.5 dev tun11 metric 1 table 10 ip rule add from 10.173.1.0/24 table 10 ip rule add to 10.173.1.0/24 table 10 ip rule add from 192.168.1.253 table 10
Now, since I also have devices getting their DNS servers from DHCP, we'll also need to add rules for those internal devices to make use of the ISPs default route to access DNS servers. To do that, the rules simply address traffic to and from the internal client ip to the DNS servers via the “main” table instead of our new table “10”:
ip rule add from 192.168.1.253 to 89.150.129.10 lookup main ip rule add from 192.168.1.253 to 89.150.129.22 lookup main ip rule add to 192.168.1.253 from 89.150.129.10 lookup main ip rule add to 192.168.1.253 from 89.150.129.22 lookup main
scripting the routes and rules
Disclaimer: I have never made use of awk before doing this configuration, so this may be the dumbest awk code ever written. It works, so that's good enough for me.
A basic script to set the routes in my case looks like this:
vpndhosts="192.168.1.253"
dnsservers="89.150.129.10 89.150.129.22"
tun_iface=$(ifconfig|grep tun|awk '{ print $1 }')
tun_inet=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $2 }'|awk -F ' ' '{ print $1 }')
tun_ptp=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $3 }'|awk -F ' ' '{ print $1 }')
tun_gw=$(echo $tun_inet |awk -F '.' '{print $1"."$2"."$3".1"}')
tun_net=$(echo $tun_inet |awk -F '.' '{print $1"."$2"."$3".0/24"}')
ip route add 0.0.0.0/1 via $tun_ptp dev $tun_iface table 10
ip route add 128.0.0.0/1 via $tun_ptp dev $tun_iface table 10
ip route add $tun_gw via $tun_ptp dev $tun_iface metric 1 table 10
ip rule add from $tun_net table 10
ip rule add to $tun_net table 10
for host in $vpndhosts;do ip rule add from $host table 10;done
for host in $vpndhosts;do for server in $dnsservers;do ip rule add from $host to $server lookup main;ip rule add from $server to $host lookup main;done;done
To add other hosts, vpndhosts is a space delimited list of local ip's.
Unfortunately, getting port-forwarding working from my chosen provider is another issue. Optware here I come.
To delete all custom rules and return to default (in my case at least):
for rule in $(ip rule list |grep -v "all lookup"|awk -F ":" '{ print $1 }');do ip rule delete pref $rule;done
So, add the two code sections above to /tmp/vpnrouteup.sh and /tmp/vpnroutedown.sh and chmod them via the Administration→Scripts→Init script:
cat > /tmp/vpnrouteup.sh <<EOF
#!/bin/sh
vpndhosts="192.168.1.253"
dnsservers="89.150.129.10 89.150.129.22"
tun_iface=\$(ifconfig|grep tun|awk '{ print \$1 }')
tun_inet=\$(ifconfig \$tun_iface|grep P-t-P|awk -F ':' '{ print \$2 }'|awk -F ' ' '{ print \$1 }')
tun_ptp=\$(ifconfig \$tun_iface|grep P-t-P|awk -F ':' '{ print \$3 }'|awk -F ' ' '{ print \$1 }')
tun_gw=\$(echo \$tun_inet |awk -F '.' '{print \$1"."\$2"."\$3".1"}')
tun_net=\$(echo \$tun_inet |awk -F '.' '{print \$1"."\$2"."\$3".0/24"}')
ip route add 0.0.0.0/1 via \$tun_ptp dev \$tun_iface table 10
ip route add 128.0.0.0/1 via \$tun_ptp dev \$tun_iface table 10
ip route add \$tun_gw via \$tun_ptp dev \$tun_iface metric 1 table 10
ip rule add from \$tun_net table 10
ip rule add to \$tun_net table 10
for host in \$vpndhosts;do ip rule add from \$host table 10;done
for host in \$vpndhosts;do for server in \$dnsservers;do ip rule add from \$host to \$server lookup main;ip rule add from \$server to \$host lookup main;done;done
EOF
cat > /tmp/vpnroutedown.sh <<EOF
#!/bin/sh
for rule in \$(ip rule list |grep -v "all lookup"|awk -F ":" '{ print \$1 }');do ip rule delete pref \$rule;done
EOF
chmod 700 /tmp/vpnrouteup.sh
chmod 700 /tmp/vpnroutedown.sh
troubleshooting
bzzt.
Port Forwarding with PIA
Here's the link with basic info.
Setup Optware via the very handy tutorial here.
Install curl via ipkg install libcurl.
Add the following to Administration→Scripts→Init:
#Setup port forwarding client-ID head -n 100 /dev/urandom | md5sum | tr -d " -" > /tmp/pia_client_id
Setup a scheduled task that runs at least once per hour that contains:
pia_client_id=$(cat /tmp/pia_client_id)
pia_user=$(head -1 /tmp/pia.txt)
pia_pw=$(tail -1 /tmp/pia.txt)
tun_inet=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $2 }'|awk -F ' ' '{ print $1 }')
curl -d "user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" https://www.privateinternetaccess.com/vpninfo/port_forward_assignment >> /tmp/port_forward.txt
What's missing is the traffic needs to go out and back over the tunnel- ip rule add?
root@wifi1:/tmp/home/root# ip rule add to 75.52.9.107 table 10 root@wifi1:/tmp/home/root# ip rule add from 75.52.9.107 table 10 root@wifi1:/tmp/home/root# ip rule add to 209.222.18.222 table 10 root@wifi1:/tmp/home/root# ip rule add from 209.222.18.222 table 10 root@wifi1:/tmp/home/root# traceroute 209.222.18.222 traceroute to 209.222.18.222 (209.222.18.222), 30 hops max, 38 byte packets 1 10.127.1.1 (10.127.1.1) 64.245 ms 64.217 ms 64.661 ms 2 buc-ird-14sw.voxility.net (93.115.85.33) 64.751 ms 64.891 ms 64.757 ms 3 buc-ird-02c.voxility.net (93.115.87.29) 64.335 ms 64.456 ms 75.212 ms 4 buc-ird-01gw.voxility.net (109.163.234.53) 70.970 ms buc-ird-04gw.voxility.net (195.60.77.13) 65.213 ms root@wifi1:/tmp/home/root# curl -d "user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" https://www.privateinternetaccess.com/vpninfo/port_ forward_assignment curl: (35) error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm
Hmmmmmmmmmm. Apparently this means curl has an SSL issue.
Ran ipkg install wget-ssl, then /opt/bin/wget –post-data=“user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet” –no-check-certificate https://ww
w.privateinternetaccess.com/vpninfo/port_forward_assignment and got:
2014-02-19 21:13:51 (633 KB/s) - `port_forward_assignment' saved [57]
root@wifi1:/tmp/home/root# more port_forward_assignment
{"error":"port forwarding not available for this region"}
root@wifi1:/tmp/home/root#
Looks like somehow my ip rule add's got dropped or aren't working? Indeed a traceroute shows traffic heading out the wrong interface.
Tomorrow…
I wrote down the wrong IP somehow and then had been copy & pasting that into my rules:
root@wifi1:/tmp/home/root# ip rule add to 75.52.9.107 lookup 10 root@wifi1:/tmp/home/root# ip rule add from 75.52.9.107 lookup 10 root@wifi1:/tmp/home/root# traceroute www.privateinternetaccess.com traceroute to www.privateinternetaccess.com (72.52.9.107), 30 hops max, 38 byte packets 1 89.150.184.1 (89.150.184.1) 22.198 ms 22.421 ms 22.421 ms 2 te1-1-0.mr1.fb.dk.ip.fullrate.dk (90.185.4.138) 19.302 ms 19.245 ms 19.104 ms 3 xe-7-3-0-0.boanqp7.dk.ip.tdc.net (188.180.68.81) 19.264 ms xe-3-0-0-0.boanqp7.dk.ip.tdc.net (195.215.109.229) 19.227 ms xe-7-3-0-0.boanqp7.dk.ip.tdc.net (188.180.68.81) 19.556 ms 4 ae4-0.asd9nqp1.nl.ip.tdc.net (83.88.30.13) 38.615 ms 37.569 ms 38.085 ms 5 amsix2.teleglobe.net (195.69.145.167) 38.603 ms 37.433 ms 37.635 ms 6 root@wifi1:/tmp/home/root# ip rule add from 72.52.9.107 lookup 10 root@wifi1:/tmp/home/root# ip rule add to 72.52.9.107 lookup 10 root@wifi1:/tmp/home/root# traceroute www.privateinternetaccess.com traceroute to www.privateinternetaccess.com (72.52.9.107), 30 hops max, 38 byte packets 1 10.115.51.1 (10.115.51.1) 67.864 ms 117.204 ms 66.705 ms 2 buc-ird-01c.voxility.net (93.114.43.193) 66.965 ms 67.535 ms 66.999 ms 3 buc-ird-01gw.voxility.net (109.163.235.49) 67.726 ms buc-ird-02gw.voxility.net (109.163.235.57) 123.000 ms buc-ird-04gw.voxility.net (109.163.235.61) 67.645 ms 4
Fixed. The return data now is simply blank:
root@wifi1:/tmp/home/root# /opt/bin/wget --post-data="user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" --no-check-certificate https://www.privateinternetaccess.com/vpninfo/port_forward_assignment
--2014-02-20 07:53:49-- https://www.privateinternetaccess.com/vpninfo/port_forward_assignment
Resolving www.privateinternetaccess.com (www.privateinternetaccess.com)... 72.52.9.107
Connecting to www.privateinternetaccess.com (www.privateinternetaccess.com)|72.52.9.107|:443... connected.
WARNING: cannot verify www.privateinternetaccess.com's certificate, issued by `/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2':
Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `port_forward_assignment'
[ <=> ] 2 --.-K/s in 0s
2014-02-20 07:53:51 (23.3 KB/s) - `port_forward_assignment' saved [2]
root@wifi1:/tmp/home/root# more port_forward_assignment
{}
Just a bit of duh I'm guessing, I was using variables in that command that this morning are all un-set:
root@wifi1:/tmp/home/root# pia_user=$(head -1 /tmp/pia.txt)
root@wifi1:/tmp/home/root# pia_pw=$(tail -1 /tmp/pia.txt)
root@wifi1:/tmp/home/root# pia_client_id=$(head -1 /tmp/pia_client_id)
root@wifi1:/tmp/home/root# tun_inet=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $2 }'|awk -F ' ' '{ print $1 }')
root@wifi1:/tmp/home/root# /opt/bin/wget --post-data="user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" --no-check-certificate -q -O - https://www.privateinternetaccess.com/vpninfo/port_forward_assignment
{"port":22172}root@wifi1:/tmp/home/root#
Ta-da! So up next is setting a port forward rule in the router along with testing a connection to the LAN client port. I just realize that my “ip rule add from” may not be good enough to get unprompted incoming traffic through to the LAN client, probably need an “ip rule add to”. Then of course there's also the on-the-fly update to transmission on the NAS to tell it what the listening port should be, or can I just update the port forward rule with the external port set to the pre-configured internal listening port? Probably that.
And in this testing I find I can't really seem to connect to/from anything via the LAN ip. I get no route to host if there's no “ip rule to” rule in place, if I add that rule, ICMP stop working. Blech.
On a hunch, restarted everything, and with the default setup, and connections out from the LAN client worked just fine. I might have messed some rules up along the way.
root@wifi1:/tmp/home/root# ip rule add to 72.52.9.107 lookup 10
root@wifi1:/tmp/home/root# /opt/bin/wget --post-data="user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" --no-check-certificate -q -O - https://www.privateinternetaccess.com/vpninfo/port_forward_assignment
{"port":22172}
Okay, and confirmed via use of ipkg installed netcat on the router that the forwarded port itself does function, so it's probably just a matter of getting iptables w/ip route to do the right thing to pass connections on to the LAN client.
[root@ragingdouche ~]# nc -v 37.221.165.196 22172 Connection to 37.221.165.196 22172 port [tcp/*] succeeded! tet ^C
root@wifi1:/tmp/home/root# /opt/bin/nc -l -p 22172 tet ^C
Found some iptables possibilities here, http://strongvpn.com/forum/viewtopic.php?id=3174:
iptables -t nat -A PREROUTING -p <tcp OR udp> --dport <port you want to forward> -j DNAT --to-destination <IP you want to forward to> iptables -A FORWARD -s <your TUN IP> -p <tcp OR udp> --dport <port you want to forward> -j ACCEPT
After playing with variations on those commands I don't get connection refused from the outside, it times out instead.
If I delete all the PREROUTING rules via iptables -t nat -D PREROUTING #rule I can at least get to the tun interface listening and responding via netcat:
root@wifi1:/tmp/home/root# /opt/bin/nc -v -l -p 22172 listening on [any] 22172 ... connect to [10.106.1.6] from li245-253.members.linode.com [173.255.235.253] 56825 test
With http://www.dd-wrt.com/phpBB2/viewtopic.php?p=782506 - GOT IT!!!
root@wifi1:/tmp/home/root# iptables -t nat -I PREROUTING -p tcp -i tun11 --dport 22172 -j DNAT --to 192.168.33.253:1024 root@wifi1:/tmp/home/root# iptables -I FORWARD -p tcp -d 192.168.33.253 --dport 1024 -j ACCEPT
On the NAS:
-bash-3.2# while : ;do ./nc -v -l -p 1024;done Connection from 173.255.235.253:60984 test
From an external host:
[root@ragingdouche ~]# nc -v 37.221.165.196 22172 Connection to 37.221.165.196 22172 port [tcp/*] succeeded! test
So- the vpnrouteup.sh script will need to set rules to access www.pia.com for port forward requests, and then the port forward script (/opt/etc/port_forward_update.sh) will need to get all the PIA client info together, get the IP of the www host, add rules to table 10 for that host, make the port forwarding request, retrieve the port, then run those last two iptables commands to forward the port to the NAS appropriately. Woot.
Current contents of port_forward_update.sh:
#!/bin/sh
#to-do: get current nat preroute rule and compare port, if it's the same do nothing, if it's different, delete the old and add the new
pia_user=$(head -1 /tmp/pia.txt)
pia_pw=$(tail -1 /tmp/pia.txt)
pia_client_id=$(head -1 /tmp/pia_client_id)
tun_inet=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $2 }'|awk -F ' ' '{ print $1 }')
#get our forwarding port from PIA
forwarded_port=$(/opt/bin/wget --post-data="user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" \
--no-check-certificate \
-q -O - https://www.privateinternetaccess.com/vpninfo/port_forward_assignment \
| awk -F ':' '{ print $2 }'| awk -F '}' '{ print $1 }')
echo $forwarded_port
#set forwarding to our NAS host
#iptables -t nat -I PREROUTING -p tcp -i tun11 --dport $forwarded_port -j DNAT --to 192.168.33.253:1024
#iptables -I FORWARD -p tcp -d 192.168.33.253 --dport 1024 -j ACCEPT
