6.14. Troubleshooting IPsec configurations

Problems with VPN connections often occur due to mismatched configurations between the endpoints.

To confirm that an IPsec connection is established, enter:

# ipsec trafficstatus
006 #8: "vpn.example.com"[1] 192.0.2.1, type=ESP, add_time=1595296930, inBytes=5999, outBytes=3231, id='@vpn.example.com', lease=198.51.100.1/32

For a successful connection, the command shows an entry with the connection’s name and details. If the output is empty, the tunnel is not established.

Diagnose IPsec connection failures caused by firewall-related problems.

If a firewall on an endpoint or an intermediate route drops Internet Key Exchange version 2 (IKEv2) packets, the ipsec up command times out and shows repeated retransmission messages:

181 "vpn.example.com"[1] 192.0.2.2 #15: initiating IKEv2 IKE SA
181 "vpn.example.com"[1] 192.0.2.2 #15: STATE_PARENT_I1: sent v2I1, expected v2R1
010 "vpn.example.com"[1] 192.0.2.2 #15: STATE_PARENT_I1: retransmission; will wait 0.5 seconds for response
010 "vpn.example.com"[1] 192.0.2.2 #15: STATE_PARENT_I1: retransmission; will wait 1 seconds for response
010 "vpn.example.com"[1] 192.0.2.2 #15: STATE_PARENT_I1: retransmission; will wait 2 seconds for response
...

You can use the tcpdump utility to capture traffic and verify if IKE or IPsec packets are dropped by a firewall, for example:

# tcpdump -i <interface> -n -n esp or udp port 500 or udp port 4500 or tcp port 4500

Note that tcpdump is of limited use for diagnosing other types of IPsec problems because the IKE protocol is encrypted.

VPN connections fail if the endpoints are not configured with matching Internet Key Exchange (IKE) versions, algorithms, IP address ranges, or pre-shared keys (PSK). If you identify a mismatch, you must align the settings on both endpoints to resolve the issue.

Diagnose intermittent IPsec connection failures caused by Maximum Transmission Unit (MTU) issues. Encryption increases packet size, leading to fragmentation and lost data when packets exceed the network’s MTU, often seen with larger data transfers.

A common symptom is that small packets, for example pings, work correctly, but larger packets, such as an SSH session, freeze after the login. To fix the problem, lower the MTU for the tunnel by adding the mtu=1400 option to the configuration file.

Resolve NAT conflicts that occur when an IPsec host also acts as a NAT router. Incorrect NAT application can translate source IP addresses before encryption, causing packets to be sent unencrypted over the network.

For example, if the source IP address of the packet is translated by a masquerade rule before IPsec encryption is applied, the packet’s source no longer matches the IPsec policy, and Libreswan sends it unencrypted over the network.

To solve this problem, add a firewall rule that excludes traffic between the IPsec subnets from NAT. This rule should be inserted at the beginning of the POSTROUTING chain to ensure it is processed before the general NAT rule.

# nft add table ip nat
# nft add chain ip nat postrouting { type nat hook postrouting priority 100 \; }
# nft add rule ip nat postrouting ip saddr 192.0.2.0/24 ip daddr 198.51.100.0/24 return

Troubleshoot kernel-level IPsec issues when a VPN tunnel appears established but no traffic flows. In this case, inspect the kernel’s IPsec state to check if the tunnel policies and cryptographic keys were correctly installed.

This process involves checking two components:

First, check if the correct policy exists in the SPD:

# ip xfrm policy
src 192.0.2.1/32 dst 10.0.0.0/8
	dir out priority 666 ptype main
	tmpl src 198.51.100.13 dst 203.0.113.22
		proto esp reqid 16417 mode tunnel

The output should contain the policies matching your leftsubnet and rightsubnet parameters with both in and out directions. If you do not see a policy for your traffic, Libreswan failed to create the kernel rule, and traffic is not encrypted.

If the policy exists, check if it has a corresponding set of keys in the SAD:

# ip xfrm state
src 203.0.113.22 dst 198.51.100.13
	proto esp spi 0xa78b3fdb reqid 16417 mode tunnel
	auth-trunc hmac(sha1) 0x3763cd3b... 96
	enc cbc(aes) 0xd9dba399...

If a policy exists but you see no corresponding state with the same reqid, it typically means the Internet Key Exchange (IKE) negotiation failed. The two VPN endpoints could not agree on a set of keys.

For more detailed diagnostics, use the -s option with either of the commands. This option adds traffic counters, which can help you identify if the kernel processes packets by a specific rule.

A defect in the kernel’s IPsec subsystem can cause it to lose sync with the IKE daemon. This can lead to discrepancies between negotiated security associations and actual IPsec policy enforcement, disrupting secure network communication.

To check for kernel-level errors, display the transform (XFRM) statistics:

# cat /proc/net/xfrm_stat

If any of the counters in the output, such as XfrmInError, show a nonzero value, it indicates a problem with the kernel subsystem. In this case, open a support case, and attach the output of the command along with the corresponding IKE logs.

Display Libreswan logs to diagnose and troubleshoot IPsec service events and issues. Access the journal for the ipsec service to gain insights into connection status and potential problems.

To display the journal, enter:

# journalctl -xeu ipsec

If the default logging level does not provide enough details, enable comprehensive debug logging by adding the following settings to the config setup section in the /etc/ipsec.conf file:

plutodebug=all
logfile=/var/log/pluto.log

Because debug logging can produce many entries, redirecting the messages to a dedicated log file can prevent the journald and systemd services from rate-limiting the messages.