The relationship between IPsec and NAT can be pretty confusing because there are more than a couple of "gotchas" hiding in amongst the acronyms. The two main gotchas are that the two main IPSec protocols have issues with most technologies that attempt to modify fields in the layer 3 and layer 4 headers. Specifically, Authentication Header (AH) doesn't work with Network Address Translation (NAT) or Port Address Translation (PAT) and that Encapsulating Security Payload (ESP) is also incompatible with PAT. Although some workarounds were discussed in the last tip about transparent tunneling, some readers may be interested in more information about why it's broken in the first place.
To understand this, it's helpful to begin by recognizing that AH's goal is to ensure that packets arrive at their destination exactly the same as they left and it uses a checksum to accomplish this. It is important to note that this checksum is applied to the entire packet, including IP headers.
Although NAT isn't the malicious attacker that AH was designed to thwart, it remains largely thwarted. As packets travel through a NAT process, NAT changes the IP address in the source or destination field and sends the packet on its way. When it arrives, the checksum is calculated on the packet by AH again, and of course, it doesn't match the existing one that was sent with the packet. Thus, the packet is rejected.
ESP's goal is quite different, and actually much more common.
Requires Free Membership to View
SearchEnterpriseWAN.com members gain immediate and unlimited access to breaking industry news, best practices for designing and managing Wide Area Networks, WAN Security, and more -- all at no cost. Join me on SearchEnterpriseWAN.com today!
Kate Gerwig, Editorial Director
ESP encrypts the data in the packet so that it cannot easily be viewed as it crosses the network. Encrypting the IP headers isn't practical, because if they were encrypted, no router in the network would be able to tell where the packet was coming from or going to. Thus, ESP headers are inserted between the layer 3 and 4 headers and only the payload of the IP packet (which includes the TCP or UDP headers, if applicable) are encrypted.
This means that if NAT fudges an IP address, it's really no big deal, because unlike AH, there's no checksum guarding the IP headers. However, if an encrypted packet travels through a PAT process, not only the IP address is changed, but also the TCP or UDP port numbers. These, unfortunately, are encrypted. The PAT process can't see them or change them. In fact, it doesn't even recognize that TCP or UDP are in use because the IP protocol field will be listed as Protocol 50, which is ESP, instead of 6 or 17, which are TCP and UDP respectively.
Thomas Alexander Lancaster IV is a consultant and author with over 15 years experience in the networking industry, focused on Internet infrastructure.
This was first published in August 2006