Routing issue between LAN and OPT1 when IPSEC enabled

[Oliver Hansen]

I have hub and spoke VPN network setup. 192.168.1.0/24 is the hub (central
office) and 192.168.x.0/24 are all the spokes (remote offices). These are
all connected with IPSEC VPN connections running a mix of linksys vpn
routers and pfSense 1.2.3-RC3. The problem I am having is related to two
pfSense boxes running 1.2.3-RC3 (I'll update to RELEASE if that's really the
problem but I'd rather wait a while). In most locations there is a single
subnet at the remote offices and that works fine. The remote offices are all
able to communicate to each other through the central office because on
their routers the IPSEC remote subnet is 192.168.0.0/16.

Here is the problem: at one location we have both 192.168.2.0/24 on LAN and
192.168.50.0/24 on OPT1. We have a VPN connection from the LAN to the hub
office and that worked fine but neither computers on the 192.168.2.0/24 or
the 192.168.1.0/24 could reach the 192.168.50.0/24 subnet. I determined that
the reason must be that any packets from the LAN must be getting sent over
the VPN tunnel before the router would check to see that it held that subnet
on one of it's own interfaces.

Just last week, I set up a second VPN tunnel between the two routers. This
one has the destination subnet of 192.168.50.0/24 and now from the hub
router we can reach that subnet but from the 192.168.2.0/24 still cannot
reach it. My thinking was that the router with LAN and OPT1 would either
route between the two subnets and if not, it would send data up one VPN
connection because it was "interesting traffic" and then it would get sent
back down the 2nd tunnel to the other subnet. Neither of these things is
happening.

Any ideas on how to get this working? If there are any details I missed,
please let me know and I will try to clarify.

[Chris Buechler]

Post by Oliver Hansen
I have hub and spoke VPN network setup. 192.168.1.0/24 is the hub (central
office) and 192.168.x.0/24 are all the spokes (remote offices). These are
all connected with IPSEC VPN connections running a mix of linksys vpn
routers and pfSense 1.2.3-RC3. The problem I am having is related to two
pfSense boxes running 1.2.3-RC3 (I'll update to RELEASE if that's really the
problem but I'd rather wait a while). In most locations there is a single
subnet at the remote offices and that works fine. The remote offices are all
able to communicate to each other through the central office because on
their routers the IPSEC remote subnet is 192.168.0.0/16.
Here is the problem: at one location we have both 192.168.2.0/24 on LAN and
192.168.50.0/24 on OPT1. We have a VPN connection from the LAN to the hub
office and that worked fine but neither computers on the 192.168.2.0/24 or
the 192.168.1.0/24 could reach the 192.168.50.0/24 subnet. I determined that
the reason must be that any packets from the LAN must be getting sent over
the VPN tunnel before the router would check to see that it held that subnet
on one of it's own interfaces.
Just last week, I set up a second VPN tunnel between the two routers. This
one has the destination subnet of 192.168.50.0/24 and now from the hub
router we can reach that subnet but from the 192.168.2.0/24 still cannot
reach it. My thinking was that the router with LAN and OPT1 would either
route between the two subnets and if not, it would send data up one VPN
connection because it was "interesting traffic" and then it would get sent
back down the 2nd tunnel to the other subnet. Neither of these things is
happening.

That traffic is going out IPsec because IPsec always wins over
anything in the system routing table including other directly attached
networks (just how it works in the FreeBSD kernel). You either have to
not include that other local subnet within your remote IPsec
definition, or use OpenVPN which will work properly in that scenario.

---------------------------------------------------------------------
To unsubscribe, e-mail: support-unsubscribe-***@public.gmane.org
For additional commands, e-mail: support-help-***@public.gmane.org

Commercial support available - https://portal.pfsense.org

[Oliver Hansen]

--snip--

Post by Oliver Hansen
Just last week, I set up a second VPN tunnel between the two routers.

This

Post by Oliver Hansen
one has the destination subnet of 192.168.50.0/24 and now from the hub
router we can reach that subnet but from the 192.168.2.0/24 still cannot
reach it. My thinking was that the router with LAN and OPT1 would either
route between the two subnets and if not, it would send data up one VPN
connection because it was "interesting traffic" and then it would get

sent

Post by Oliver Hansen
back down the 2nd tunnel to the other subnet. Neither of these things is
happening.

That traffic is going out IPsec because IPsec always wins over
anything in the system routing table including other directly attached
networks (just how it works in the FreeBSD kernel). You either have to
not include that other local subnet within your remote IPsec
definition, or use OpenVPN which will work properly in that scenario.

Thanks for the reply. I can understand that IPsec always wins but why if it
is getting sent up the VPN tunnel does it not get sent back down the second
VPN tunnel to the 192.168.50.0/24 subnet? Any of my other networks such as
192.168.3.0/24 can send traffic to the .50 network and receive replies. Is
there something about having two IPsec VPNs between the same two boxes that
causes this not to work?

Example A: 192.168.3.0/24 -------------> 192.168.1.0/24 ------------->
192.168.50.0/24 = successful
Example B: 192.168.2.0/24 -------------> 192.168.1.0/24 -----------X
192.168.50.0/24 = no success

[Yehuda Katz]

Sounds to me like a NAT Reflection issue

Post by Oliver Hansen

--snip--

Post by Oliver Hansen
Just last week, I set up a second VPN tunnel between the two routers.

This

Post by Oliver Hansen
one has the destination subnet of 192.168.50.0/24 and now from the hub
router we can reach that subnet but from the 192.168.2.0/24 still

cannot

Post by Oliver Hansen
reach it. My thinking was that the router with LAN and OPT1 would either
route between the two subnets and if not, it would send data up one VPN
connection because it was "interesting traffic" and then it would get

sent

Post by Oliver Hansen
back down the 2nd tunnel to the other subnet. Neither of these things is
happening.

That traffic is going out IPsec because IPsec always wins over
anything in the system routing table including other directly attached
networks (just how it works in the FreeBSD kernel). You either have to
not include that other local subnet within your remote IPsec
definition, or use OpenVPN which will work properly in that scenario.

Thanks for the reply. I can understand that IPsec always wins but why if it
is getting sent up the VPN tunnel does it not get sent back down the second
VPN tunnel to the 192.168.50.0/24 subnet? Any of my other networks such as
192.168.3.0/24 can send traffic to the .50 network and receive replies. Is
there something about having two IPsec VPNs between the same two boxes that
causes this not to work?
Example A: 192.168.3.0/24 -------------> 192.168.1.0/24 ------------->
192.168.50.0/24 = successful
Example B: 192.168.2.0/24 -------------> 192.168.1.0/24 -----------X
192.168.50.0/24 = no success

[Oliver Hansen]

Post by Yehuda Katz
Sounds to me like a NAT Reflection issue

Post by Oliver Hansen

--snip--

Post by Oliver Hansen
Just last week, I set up a second VPN tunnel between the two routers.

This

Post by Oliver Hansen
one has the destination subnet of 192.168.50.0/24 and now from the hub
router we can reach that subnet but from the 192.168.2.0/24 still

cannot

Post by Oliver Hansen
reach it. My thinking was that the router with LAN and OPT1 would

either

Post by Oliver Hansen
route between the two subnets and if not, it would send data up one VPN
connection because it was "interesting traffic" and then it would get

sent

Post by Oliver Hansen
back down the 2nd tunnel to the other subnet. Neither of these things

is

Post by Oliver Hansen
happening.

That traffic is going out IPsec because IPsec always wins over
anything in the system routing table including other directly attached
networks (just how it works in the FreeBSD kernel). You either have to
not include that other local subnet within your remote IPsec
definition, or use OpenVPN which will work properly in that scenario.

Thanks for the reply. I can understand that IPsec always wins but why if
it is getting sent up the VPN tunnel does it not get sent back down the
second VPN tunnel to the 192.168.50.0/24 subnet? Any of my other networks
such as 192.168.3.0/24 can send traffic to the .50 network and receive
replies. Is there something about having two IPsec VPNs between the same two
boxes that causes this not to work?
Example A: 192.168.3.0/24 -------------> 192.168.1.0/24 ------------->
192.168.50.0/24 = successful
Example B: 192.168.2.0/24 -------------> 192.168.1.0/24 -----------X
192.168.50.0/24 = no success

I'm not sure how NAT reflection would be involved. My understanding of NAT

reflection is that is enables users to access resources on the WAN IP that
they are NATted through. This is not the case.

It sounds like OpenVPN would be an easier fit here and had I known that
before setting up all the connections I would have started with that. I am
still trying to learn what is causing the one subnet not be able to reach
the other through the VPN though when all other subnets can do this through
IPsec.

[Trevor Benson]

Had the same issue you describe, as I have multiple ranges at my office that I connect to from my home network.

I started to dump traffic on both firewalls enc0 interface, and the network traffic that works appears on both interfaces. The ICMP traffic that does not reach the other end does not appear on either enc0 interface. I already know about creating custom pre-shared keys if you have multiple connections between the same networks (so if you have 2 tunnels between the same 2 endpoints and use PSK, each tunnel will require its own key and could cause issues similar to what your seeing).

Home office networks:
I have 10.101.0.0/24 and 10.101.1.0/24 on my home pfsense firewall, represented as 10.101.0.0/23.

Main office networks:
I have my office LAN of 10.11.200.0/24. I have various networks for DMZ (each a respective /24), LAB, and VoIP, represented by 172.16.0.0/14 (covering 172.16-172.19, and all of the additional /24's those include.


I can ping from 10.101.0.0/23 into the 172.16.0.0/14 networks. Traffic can be destined into my network as well from the 172.16.0.0/14 networks.

I cannot ping from 10.101.0.0/23 into 10.11.200.0/24 and 10.11.200.0/24 cannot reach my network. Watching the traffic on enc0 shows that neither side actually forwards anything over the encrypted interface when these two networks are trying to reach eachother. If i dump the lan interface i can see the packet reaching the firewall, but it doesnt appear to leave the WAN, or IPSEC interfaces.

SAD and SPD were there, status showed both tunnels as active, but only one passed traffic. Further testing showed even stranger results. I could ping into the 172.16.0.0/14 networks, hence i could also make a voip call. BUT when my employees tried to reach me, they got dead air, and not traffic reach my end to initiate the SIP negotiation. So now it appeared i had unidirectional access between the networks. Before i tested the alternate direction i tried recreating the tunnels from scratch, still had the same results. I decided to recreate the tunnels from scratch to test, and found the same result. I deleted the 10.11.200.0/24 tunnels from both sides, and traffic flows both directions now over the single tunnel. It started to sound like IPSEC rule generation issues so i went to Firewall rules and created matching rules to the flows, and now traffic is working normally. I already had a any protocol, any source to any destination on any port rule for testing. However, adding the specific rulesets seemed to resolve the issue.

I did not however check pfctl to see if the rules were existing already or if they got munged somehow. I should be labbing up some systems to test some XMLRPC issues i had in RC3, so ill see if i can get some time to test the IPSEC multiple tunnels again and see if it exhibits the same symptoms.


Trevor Benson
A1 Networks | Network Engineer
dCAP- Digium Certified Asterisk Professional
LPIC-1, Network+, CNA, MCP
DID [phone]
Fax [phone]

Post by Yehuda Katz
Sounds to me like a NAT Reflection issue
--snip--

Post by Oliver Hansen
Just last week, I set up a second VPN tunnel between the two routers. This
one has the destination subnet of 192.168.50.0/24 and now from the hub
router we can reach that subnet but from the 192.168.2.0/24 still cannot
reach it. My thinking was that the router with LAN and OPT1 would either
route between the two subnets and if not, it would send data up one VPN
connection because it was "interesting traffic" and then it would get sent
back down the 2nd tunnel to the other subnet. Neither of these things is
happening.

That traffic is going out IPsec because IPsec always wins over
anything in the system routing table including other directly attached
networks (just how it works in the FreeBSD kernel). You either have to
not include that other local subnet within your remote IPsec
definition, or use OpenVPN which will work properly in that scenario.
Thanks for the reply. I can understand that IPsec always wins but why if it is getting sent up the VPN tunnel does it not get sent back down the second VPN tunnel to the 192.168.50.0/24 subnet? Any of my other networks such as 192.168.3.0/24 can send traffic to the .50 network and receive replies. Is there something about having two IPsec VPNs between the same two boxes that causes this not to work?
Example A: 192.168.3.0/24 -------------> 192.168.1.0/24 -------------> 192.168.50.0/24 = successful
Example B: 192.168.2.0/24 -------------> 192.168.1.0/24 -----------X 192.168.50.0/24 = no success
I'm not sure how NAT reflection would be involved. My understanding of NAT reflection is that is enables users to access resources on the WAN IP that they are NATted through. This is not the case.
It sounds like OpenVPN would be an easier fit here and had I known that before setting up all the connections I would have started with that. I am still trying to learn what is causing the one subnet not be able to reach the other through the VPN though when all other subnets can do this through IPsec.

[Oliver Hansen]

Post by Trevor Benson
Had the same issue you describe, as I have multiple ranges at my office
that I connect to from my home network.
I started to dump traffic on both firewalls enc0 interface, and the network
traffic that works appears on both interfaces. The ICMP traffic that does
not reach the other end does not appear on either enc0 interface. I already
know about creating custom pre-shared keys if you have multiple connections
between the same networks (so if you have 2 tunnels between the same 2
endpoints and use PSK, each tunnel will require its own key and could cause
issues similar to what your seeing).
I have 10.101.0.0/24 and 10.101.1.0/24 on my home pfsense firewall,
represented as 10.101.0.0/23.
I have my office LAN of 10.11.200.0/24. I have various networks for DMZ
(each a respective /24), LAB, and VoIP, represented by 172.16.0.0/14(covering 172.16-172.19, and all of the additional /24's those include.
I can ping from 10.101.0.0/23 into the 172.16.0.0/14 networks. Traffic
can be destined into my network as well from the 172.16.0.0/14 networks.
I cannot ping from 10.101.0.0/23 into 10.11.200.0/24 and 10.11.200.0/24cannot reach my network. Watching the traffic on enc0 shows that neither
side actually forwards anything over the encrypted interface when these two
networks are trying to reach eachother. If i dump the lan interface i can
see the packet reaching the firewall, but it doesnt appear to leave the WAN,
or IPSEC interfaces.
SAD and SPD were there, status showed both tunnels as active, but only one
passed traffic. Further testing showed even stranger results. I could ping
into the 172.16.0.0/14 networks, hence i could also make a voip call. BUT
when my employees tried to reach me, they got dead air, and not traffic
reach my end to initiate the SIP negotiation. So now it appeared i had
unidirectional access between the networks. Before i tested the alternate
direction i tried recreating the tunnels from scratch, still had the same
results. I decided to recreate the tunnels from scratch to test, and found
the same result. I deleted the 10.11.200.0/24 tunnels from both sides,
and traffic flows both directions now over the single tunnel. It started to
sound like IPSEC rule generation issues so i went to Firewall rules and
created matching rules to the flows, and now traffic is working normally. I
already had a any protocol, any source to any destination on any port rule
for testing. However, adding the specific rulesets seemed to resolve the
issue.
I did not however check pfctl to see if the rules were existing already or
if they got munged somehow. I should be labbing up some systems to test
some XMLRPC issues i had in RC3, so ill see if i can get some time to test
the IPSEC multiple tunnels again and see if it exhibits the same symptoms.
Trevor Benson
A1 Networks | Network Engineer
dCAP- Digium Certified Asterisk Professional
LPIC-1, Network+, CNA, MCP
DID [phone]
Fax [phone]
Thanks for the information Trevor.

I deleted the 10.11.200.0/24 tunnels from both sides, and traffic flows both
directions now over the single tunnel. It started to sound like IPSEC rule
generation issues so i went to Firewall rules and created matching rules to
the flows, and now traffic is working normally.

Do you mean that with the 10.11.200.0/24 tunnel deleted from both ends that
your 10.101.0.0/23 network can reach the 10.11.200.0/24 over the IPsec VPN
to 172.16.0.0/14?

Please do let me know if you find any more information with multiple
tunnels. In the meantime I think I'll probably just have to live with the
connection issue. The one technician at that site will just connect to a
machine in my 192.168.1.0/24 network and then connect from that machine to a
machine in the 192.168.50.0/24 network.