Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

forced source based on interface (br2) #156

Open
david00769 opened this issue Jan 23, 2023 · 8 comments
Open

forced source based on interface (br2) #156

david00769 opened this issue Jan 23, 2023 · 8 comments

Comments

@david00769
Copy link

david00769 commented Jan 23, 2023 •
edited by peacey
Loading

For the life of me I can't seem to get this to work and am stuck - would appreciate any help.

Scenario - Site to Site VPN established using Unifi's open VPN feature and is tested as working. I logged on to router, and tested ping of all available interfaces (side one of tunnel is 10.8.8.1, other side is 10.8.8.2). Can also ping remote gateway and access web pages on devices at both ends of the tunnel from the router.

When I log onto the wifi network, it works (until I bring the tunnel upland then traffic stops).

I have a separate wifi network (192.168.3.0/24) on interface br2 that I would like to fource everything through from that network the default gateway on the other side of the tunnel.

Here are my logs...

# ip route show
10.0.0.0/24 via 10.8.8.2 dev tun1 
10.0.1.0/24 via 10.8.8.2 dev tun1 
10.8.8.2 dev tun1 proto kernel scope link src 10.8.8.1 
167.XXX.XXX.XXX/22 dev eth8 proto kernel scope link src 167.XXX.XXX.XXX
192.168.3.0/24 dev br2 proto kernel scope link src 192.168.3.1 
192.168.100.0/24 dev br0 proto kernel scope link src 192.168.100.1 

# iptables -t mangle -S | grep VPN
-N VPN_FORWARD
-N VPN_OUTPUT
-N VPN_POSTROUTING
-N VPN_PREROUTING
-A PREROUTING -j VPN_PREROUTING
-A FORWARD -j VPN_FORWARD
-A OUTPUT -j VPN_OUTPUT
-A POSTROUTING -j VPN_POSTROUTING
-A VPN_PREROUTING -i br2 -j MARK --set-xmark 0x169/0xffffffff
-A VPN_PREROUTING -d 10.8.8.1/32 -m mark --mark 0x169 -j MARK --set-xmark 0x0/0xffffffff
# iptables -t nat -S | grep VPN
-N VPN_POSTROUTING
-N VPN_PREROUTING
-A PREROUTING -j VPN_PREROUTING
-A POSTROUTING -j VPN_POSTROUTING
# iptables -t filter -S | grep VPN
-N VPN_FORWARD
-N VPN_INPUT
-A INPUT -j VPN_INPUT
-A FORWARD -j VPN_FORWARD
# ip6tables -t mangle -S | grep VPN
-N VPN_FORWARD
-N VPN_OUTPUT
-N VPN_POSTROUTING
-N VPN_PREROUTING
-A PREROUTING -j VPN_PREROUTING
-A FORWARD -j VPN_FORWARD
-A OUTPUT -j VPN_OUTPUT
-A POSTROUTING -j VPN_POSTROUTING
-A VPN_PREROUTING -i br2 -j MARK --set-xmark 0x169/0xffffffff
# ip6tables -t nat -S | grep VPN
-N VPN_POSTROUTING
-N VPN_PREROUTING
-A PREROUTING -j VPN_PREROUTING
-A POSTROUTING -j VPN_POSTROUTING
-A VPN_POSTROUTING -o tun1 -m mark --mark 0x169 -j MASQUERADE
# ip6tables -t filter -S | grep VPN
-N VPN_FORWARD
-N VPN_INPUT
-A INPUT -j VPN_INPUT
-A FORWARD -j VPN_FORWARD


# ip route show table 101
0.0.0.0/1 via 10.8.8.2 dev tun1 
blackhole default 
128.0.0.0/1 via 10.8.8.2 dev tun1

# ip -6 route show table 101
blackhole default dev lo metric 1024 pref medium

# ip rule
0:	from all lookup local 
99:	from all fwmark 0x169 lookup 101 
32000:	from all lookup main 
32500:	from 167.179.139.97 lookup 201 
32766:	from all lookup 201 
32767:	from all lookup default 

# ip -6 rule
0:	from all lookup local 
99:	from all fwmark 0x169 lookup 101 
32000:	from all lookup main 
32501:	from fe80::72a7:41ff:feed:ccc9 lookup 201 
32766:	from all lookup 201 

# ps | grep updown.sh
12268 root     grep updown.sh
22248 root     {updown.sh} /bin/sh /etc/split-vpn/vpn/updown.sh tun1 up site1
22251 root     {updown.sh} /bin/sh /etc/split-vpn/vpn/updown.sh tun1 up site1

Here is the config file I am using

# cat vpn.conf
### SPLIT VPN OPTIONS ###
# Enter multiple entries separated by spaces.
# Do not enter square brackets around the entries.

# Force these sources through the VPN.
# Format: [brX] for interface. [IP/nn] for IP. [xx:xx:xx:xx:xx:xx] for mac.
FORCED_SOURCE_INTERFACE="br2"
FORCED_SOURCE_IPV4=""
FORCED_SOURCE_IPV6=""
FORCED_SOURCE_MAC=""

# Format: [tcp/udp/both]-[IP/MAC Source]-[port1,port2:port3,port4,...]
# Maximum 15 ports per entry.
FORCED_SOURCE_IPV4_PORT=""
FORCED_SOURCE_IPV6_PORT=""
FORCED_SOURCE_MAC_PORT=""

# Force these destinations through the VPN. 
# These destinations will be forced regardless of source.
# Format: [IP/nn]
FORCED_DESTINATIONS_IPV4=""
FORCED_DESTINATIONS_IPV6=""

# Force local UDM traffic going out of these WAN interfaces to go through the
# VPN instead for both IPv4 and IPv6 traffic.
# This does not include routed traffic, only local traffic generated by the UDM.
# Do not enable this unless you want to force UDM local traffic through the VPN.
# For UDM-Pro, set to "eth8" for WAN1/Ethernet port, or "eth9" for WAN2/SFP+ port, 
# or "eth8 eth9" for both. For UDM Base, set to "eth1" for the WAN port.
# This option might cause unintended problems, so disable it if you encounter any issues.
FORCED_LOCAL_INTERFACE=""

# Exempt these sources from the VPN. 
# Format: [IP/nn] for IP. [xx:xx:xx:xx:xx:xx] for mac.
EXEMPT_SOURCE_IPV4=""
EXEMPT_SOURCE_IPV6=""
EXEMPT_SOURCE_MAC=""

# Format: [tcp/udp/both]-[IP/MAC Source]-[port1,port2:port3,port4,...]
# Maximum 15 ports per entry.
EXEMPT_SOURCE_IPV4_PORT=""
EXEMPT_SOURCE_IPV6_PORT=""
EXEMPT_SOURCE_MAC_PORT=""

# Exempt these destinations from the VPN. 
# Format: [IP/nn]
EXEMPT_DESTINATIONS_IPV4=""
EXEMPT_DESTINATIONS_IPV6=""

# Force/exempt these IP sets
# IP sets need to be created before this script is run or the script will error.
# IP sets can be updated externally and will be matched dynamically.
# Each IP set entry consists of the IP set name and whether to match on source
# or destination. src/dst needs to be specified for each IP set field.
#
# Enable NAT hairpin by exempting UBIOS_ADDRv4_ethX:dst for IPv4 or
# UBIOS_ADDRv6_ethX:dst for IPv6 (where X = 8 for RJ45, or 9 for SFP+ WAN).
# For IPv6 prefix delegation, exempt UBIOS_ADDRv6_brX, where X = VLAN number (0 = LAN).
#
# To allow communication with your VLAN subnets without hardcoding the subnets,
# exempt the UBIOS_NETv4_brX:dst ipset for IPv4 or UBIOS_NETv6_brX:dst for IPv6.
#
# Format: [IPSet Name]:[src/dst,src/dst,...]
FORCED_IPSETS=""
EXEMPT_IPSETS=""

# VPN port forwards.
# Format: [tcp/udp/both]-[VPN Port]-[Forward IP]-[Forward Port]
PORT_FORWARDS_IPV4=""
PORT_FORWARDS_IPV6=""

# Redirect IPv4 and IPv6 DNS to these addresses for VPN-destined traffic.
# Note that many VPN providers redirect DNS going through their VPN network
# to their own DNS servers. Redirection to other IPs might not work on all providers,
# except for DNS redirects to a local address, or rejecting DNS traffic completely.
#
# IPV4 Format: [IP] to redirect to IP, "DHCP" if using OpenVPN or OpenConnect to obtain
# DNS from DHCP options, or "REJECT" to reject all DNS traffic. "DHCP" is not supported on
# other VPN types like wireguard/external.
#
# Example: Get DNS from DHCP
DNS_IPV4_IP="DHCP"
DNS_IPV4_PORT=53
# Set this to the interface (brX) the DNS is on if it is a local IP. Leave blank for
# non-local IPs. Local DNS redirects will not work without specifying the interface.
DNS_IPV4_INTERFACE=""

# IPV6 Format: [IP] to redirect to IP, or "REJECT" to reject IPv6 DNS traffic completely.
# IPV6 Format: [IP] to redirect to IP, "DHCP" if using OpenConnect to obtain DNS from DHCP
# options, or "REJECT" to reject all DNS traffic. "DHCP" is not supported on
# other VPN types.
DNS_IPV6_IP=""
DNS_IPV6_PORT=53
DNS_IPV6_INTERFACE=""

# Bypass masquerade (SNAT) for these source IPs. This option should only be used if your 
# VPN server is setup to know how to route the subnet you do not want to masquerade 
# (e.g.: the "iroute" option in OpenVPN).
# Set these options to ALL to disable masquerading completely.
# Format: [IP/nn] or "ALL"
BYPASS_MASQUERADE_IPV4="ALL"
BYPASS_MASQUERADE_IPV6=""

# Enabling kill switch drops VPN-destined traffic that doesn't go through the VPN.
KILLSWITCH=0

# Enable this only if you are testing or you don't care about your real IP leaking
# when the vpn client restarts or exits.
REMOVE_KILLSWITCH_ON_EXIT=1

# Enable this if you added blackhole routes in the Unifi Settings to prevent Internet
# access at system startup before the VPN script runs. This option removes the blackhole 
# routes to restore Internet access after the killswitch has been enabled. 
# If you do not set this to 1, openvpn will not be able to connect at startup, and your 
# Internet access will never be enabled until you manually remove the blackhole routes. 
# Set this to 0 only if you did not add any blackhole routes.
REMOVE_STARTUP_BLACKHOLES=1

# Set the VPN provider.
# "openvpn" for OpenVPN (default), "openconnect" for OpenConnect, "external" for wireguard,
# or "nexthop" for an external VPN client.
VPN_PROVIDER="nexthop"

# If using "external" for VPN_PROVIDER, set this to the VPN endpoint IP so that the
# gateway route can be automatically added for the VPN endpoint.
# OpenVPN passes the VPN endpoint IP to the script and will override these values.
# These must be defined if using VPN_PROVIDER="nexthop".
VPN_ENDPOINT_IPV4="10.8.8.2"
VPN_ENDPOINT_IPV6=""

# Set this to the route table that contains the gateway route, "auto", or "disabled".
# The Ubiquiti route table is "201" if you're using Ethernet, "202" for SFP+, and
# "203" for U-LTE.
# Default is "auto" which works with WAN failover and automatically changes the endpoint
# via gateway route when the WAN or gateway routes changes.
# Set to "disabled" if you are using the nexthop option to connect to a VPN on your LAN.
GATEWAY_TABLE="disabled"

# Set the MSS clamping on packets going out the VPN tunnel. Usually, it is not needed to
# set this manually, but some VPN connections stall if the MSS clamping is not set correctly.
# Typical values range from 1240 to 1460, but it could be lower.
MSS_CLAMPING_IPV4=""
MSS_CLAMPING_IPV6=""

# Set this to the timer to use for the rule watcher (in seconds).
# The script will wake up every N seconds to re-add rules if they're deleted by
# the system, or change gateway routes if they changed. Default is 1 second.
WATCHER_TIMER=1

# Options for custom table and chains.
# These options need to be unique for each instance of openvpn if running multiple.
ROUTE_TABLE=101
MARK=0x169
PREFIX="VPN_"
PREF=99
DEV=tun1

# To execute commands when the VPN connects or disconnects, you can use the 
# callback functions hooks_pre_up, hooks_up, hooks_down, and 
# hooks_force_down. These functions will be invoked in response to VPN events 
# pre-up, up, down, and force-down respectively.
#
# For an example on using these hooks, please see vpn.conf.filled.sample.
#
@peacey
Copy link
Owner

peacey commented Jan 23, 2023 •
edited
Loading

Hi @david00769,

I believe your config looks correct to force devices from br2 (your wifi).

  1. What router are you using on the remote end, and what router on the local end?

  2. Did you add 192.168.3.0/24 on the remote gateway's site-to-site settings as a remote subnet?

  3. Did you make sure your WiFi devices are on br2 and getting IPs in the subnet 192.168.3.0/24?

  4. After you bring the split-vpn configuration up, can you test the following pings from a device on br2 (your WiFi device or LAN)?

    ping 1.1.1.1
ping 10.8.8.2
ping 10.0.0.1 (or some IP on 10.0.0.0/24)

  5. In your vpn.conf, you should also set MSS_CLAMPING_IPV4="1382" (might have to go even lower) or sites might stall when using a site-to-site tunnel. But the ping in (4) should succeed regardless of this setting.

@david00769 david00769 changed the title fourced source based on interface (br2) forced source based on interface (br2) Jan 24, 2023
@david00769
Copy link
Author

david00769 commented Jan 24, 2023 via email

@peacey
Copy link
Owner

peacey commented Jan 24, 2023 •
edited
Loading

Great @david00769. Glad to see that fixed it. You should also set MSS_CLAMPING_IPV4="1382" or you might run into HTTPS sites not loading or stalling (keep this in mind if you don't change the option now).

For the boot script issue, if you already installed the boot service try this to fix it. There is a bug in udm-boot that doesn't enable it automatically. You have to do it manually the first time after installation. Run these commands:

unifi-os shell
systemctl enable udm-boot
systemctl status udm-boot
exit

Once you do that, make sure 99-run-vpn.sh is in /mnt/data/on_boot.d, and make sure it has executable permissions. Then restart and see if it runs at boot.

Before you restart though, delete the openvpn.log found in the same folder as your vpn.conf. Then after you reboot, show me the output of the openvpn.log file so we can see if it did run or not.

If it still doesn't run at boot and openvpn.log wasn't created, show me the output of the following after the reboot.

unifi-os shell
systemctl status udm-boot
exit

@david00769
Copy link
Author

david00769 commented Jan 29, 2023 via email

@david00769
Copy link
Author

david00769 commented Jan 29, 2023 via email

@david00769
Copy link
Author

david00769 commented Jan 29, 2023 via email

@peacey
Copy link
Owner

peacey commented Jan 29, 2023

Hey @david00769,

From your log nothing is wrong with the boot service, it's running fine. The problem is the site-to-site interface tun1 isn't created till after this script runs, so the script is failing on that device not being found:

Jan 29 11:09:13 ubnt ssh-proxy[72]: [Sun Jan 29 11:09:13 AEDT 2023] split-vpn: t
un1 up: Loading configuration from /etc/split-vpn/nexthop/site1/vpn.conf.
Jan 29 11:09:13 ubnt ssh-proxy[72]: Cannot find device "tun1"

So what you should do is wait for tun1 to come up before running the up command. Just modify your run-vpn.sh in your nexthop folder and replace it with:

#!/bin/sh

# Load configuration and bring routes up
cd /etc/split-vpn/nexthop/target
. ./vpn.conf

while ! ip link show dev ${DEV} 1>/dev/null 2>&1; do
     sleep 5
done

/etc/split-vpn/vpn/updown.sh ${DEV} up site1

Make sure to replace the cd command near the top with the correct directory, and the site1 at the bottom with whatever nickname you were using.

That while loop should make it check for tun1 every 5 seconds until the site-to-site device is up, then it will continue with the updown command. See if that works for you.

@david00769
Copy link
Author

david00769 commented Jan 29, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@peacey @david00769