My setup becomes more and more complicated bellow there are the updated steps of my configurations with references to the original posts containing the partial configurations.
STEP 1 EdgeRouter : Dual WAN with Hair Pin
Initially I started with a dual-wan configuration with some extra setting to exclude from load balancing the hair pin connections: EdgeRouter : Dual WAN with Hair Pin
STEP 2 LAN-to-LAN Exclusion
Then because I added some new sub-networks (a wifi AP ) and I started to have issues due to the dual WAN setup I discovered that I have to exclude the local networks from the dual WAN setup.
After the modify rule was added at Step 1 (to exclude the hair pin connections) the configuration (from CLI) looked like:
ubnt@ubnt:~$ configure
[edit]
ubnt@ubnt# show firewall modify
modify balance {
rule 10 {
action modify
destination {
group {
address-group ADDRv4_eth0
}
}
modify {
table main
}
}
rule 20 {
action modify
modify {
lb-group G
}
}
}
[edit]
As instructed by the ubnt forum I had to add another rule to exclude also from the load-balancing the LAN to LAN traffic.
See the updated tutorial from ubnt for this EdgeMAX – Dual WAN Load-Balance Feature
LAN-to-LAN Exclusion
From CLI do the following:
Lower the priority of the existing rules:
configure
edit firewall modify balance
rename rule 20 to rule 30
commit
edit firewall modify balance
rename rule 10 to rule 20
commit
save
exit
Add LAN-to LAN traffic exclusion from load balancing by adding a new modify rule that has a bigger priority than the load balancing rule.
Also create a group where all the sub-nets are added as members.
configure
set firewall group network-group LAN_NETS network 192.168.1.0/24
set firewall group network-group LAN_NETS network 192.168.2.0/24
set firewall group network-group LAN_NETS network 192.168.9.0/24
commitset firewall modify balance rule 10 destination group network-group LAN_NETS
set firewall modify balance rule 10 action modify
set firewall modify balance rule 10 modify table main
commit
Note that 192.168.9.0/24 is added here because is going to be used later when we will add the openvpn site-to-site VPN.
After this the final setup looks like the following:
ubnt@ubnt:~$ configure
[edit]
ubnt@ubnt# show firewall modify
modify balance {
rule 10 {
action modify
destination {
group {
network-group LAN_NETS
}
}
modify {
table main
}
}
rule 20 {
action modify
destination {
group {
address-group ADDRv4_eth0
}
}
modify {
table main
}
}
rule 30 {
action modify
modify {
lb-group G
}
}
}
[edit]
STEP 3 EdgeRouter: OpenVPN site-to-site VPN
Another complication was added to the configuration when I added a second site to my network. This remote site has an EdgeRouter Lite so I decided to make a permanent OpenVpn site-to-site permanent VPN between sites
The setup is described in the following post: EdgeRouter: OpenVPN site-to-site VPN
STEP 4 VPN Exclusion (not yet implemented in the current firmware)
After STEP 3 was done I still had a strange issue. My VPN connection was dropping packets. I discovered that if I disabled the secondary WAN suddenly everything was OK. So I concluded that the cause of this unstable VPN was the load-balancing. In fact it was the same behaviour I noticed before STEP 2 was done when I was ping-ing between machines from different LANs.
Note that this is not yet implemented in the current firmware (1.8) but this is how I expected that should work as a solution. It was confirmed by ubiquity support that this is one of the solutions that is taken into account in a future firmware update.
See STEP 5 for the current partial solution.
From CLI do the following:
Lower the priority of the existing load-balance rule:
configure
edit firewall modify balance
rename rule 30 to rule 40
commit
save
exit
Add an exclusion rule for the vtun0 site-to-site VPN interface just before the load-balance rule.
configure
edit firewall modify balance
set rule 30 modify table main
set rule 30 destination group address-group ADDRv4_vtun0
commit
save
exit
The final configuration then looked like:
ubnt@ubnt# show firewall modify
modify balance {
rule 10 {
action modify
destination {
group {
network-group LAN_NETS
}
}
modify {
table main
}
}
rule 20 {
action modify
destination {
group {
address-group ADDRv4_eth0
}
}
modify {
table main
}
}
rule 30 {
destination {
group {
address-group ADDRv4_vtun0
}
}
modify {
table main
}
}
rule 40 {
action modify
modify {
lb-group G
}
}
}
[edit]
Then I enabled the second WAN from eth1 on my main site.
Restarted the openVPN:
reset openvpn interface vtun0
Force the reload of the load-balancing configuration by killing the process. This is a trick from UBNT-stig 🙂
Hmm “WAN Out” is empty. I wonder if you ran into a bug I fixed the last week. Can you try doing “sudo pkill ubnt-util” from the CLI and then see if “WAN Out” gets filled in?
Check if load-balance is up and acting as fail-over:
ubnt@ubnt:~$ show load-balance status
Group G
interface : eth0
carrier : up
status : active
gateway : 84.117.103.1
route table : 201
weight : 100%
flows
WAN Out : 44
WAN In : 52
Local Out : 172
interface : eth1
carrier : up
status : failover
gateway : 192.168.0.1
route table : 202
weight : 0%
flows
WAN Out : 0
WAN In : 0
Local Out : 136
Checked the status of the connection:
show openvpn status site-to-site
STEP 5 Current Issue still pending
The main issue I have now is caused by the load-balancing setup.
When both WANs are active any connection through the vtun0 is loosing half of the packages.
This is because the logic solution from STEP 4 is not yet supported by the current firmware.
After asking on the Ubiquity forums I got the following explanation in the topic I opened http://community.ubnt.com/t5/EdgeMAX/Dual-WAN-failover-OpenVPN-site-to-site/m-p/1524860/highlight/false#M104986:
UBNT-stig wrote:
I think the problem is likely that load-balance uses a separate routing table for each WAN and those routing tables only include the default gateway. Normally the way around that is to create your own routing table:
ubnt@ubnt# show protocols
static {
interface-route 192.168.9.0/24 {
next-hop-interface vtun0 {
}
}
table 1 {
interface-route 192.168.9.0/24 {
next-hop-interface vtun0 {
}
}
route 0.0.0.0/0 {
next-hop {
}
}
}
}
[edit]
And then tell load-balace to use that table.
set load-balance group G interface eth0 route table 1
Unfortunately your gateway is from dhcp and may change over time. I've been thinking of ways to work around that similar to ADDRv4_eth0, but can't use firewall groups in the routing table.
So my problem is that there is a partial solution that I cannot apply due to the fact gateway cannot be specified as DHCP in other than the default routing table.
I am still waiting for a solution that will come from Ubiquity most probably in a future firmware update. There is a thread in fact requesting for this feature http://community.ubnt.com/t5/EdgeMAX-Beta/DHCP-route-in-other-routing-table/m-p/1525360#M15702