Attach multiple IPs to a NAT Gateway to scale your egress traffic pattern
Last year, we increased the auto scaling capacity of NAT Gateways to 100 Gbps and 10 million packets per second (pps). We have also just increased NAT Gateway’s capacity to support up to 440,000 concurrent connections to a unique destination, which is eight times the previous 55,000 limit. A unique destination is recognized by a unique combination of destination IP address, destination port, and protocol. If any of these parameters change, then it’s counted as a new unique destination. Amazon Web Services NAT Gateway can now establish up to 440,000 concurrent connections to a unique destination. You can take advantage of this feature by either creating a NAT Gateway with multiple IP addresses, or by associating secondary IP addresses with an existing NAT Gateway. In this post, we explain how you can use this new feature on your NAT Gateways.
Traffic flows and Amazon Web Services NAT Gateways
First, let’s define what a traffic flow is and why it matters for the NAT Gateway. We assume you’re familiar with the TCP/IP protocol stack and header encapsulation mechanisms. A network flow, or packet flow, is generally defined by a source IP, and Port, destination IP, and Port, and protocol, for example TCP, UDP, or ICMP. You have two options when performing NAT: 1:1 NAT, and Port Address Translation (PAT). Amazon Web Services NAT Gateways performs Port Address Translation. Amazon Web Services NAT Gateways performs source-NAT, and translates the IP address of the source with its private IP address. Then, the Internet Gateway (IGW) translates the private IP address of the NAT Gateway to the Elastic IP address associated with the NAT Gateway. In Figure 1 below, we show the packet walk for two flows from an
Figure-1: NAT Gateway functionality
Amazon Web Services NAT Gateways can be deployed in a distributed model, where each VPC has its own NAT Gateways, or
Figure 2 below shows a sample deployment, with multiple workloads communicating with a public server on TCP port 389. The flows shown in Figure 2 have the same destination IP, destination port, and protocol. The source IP address of the flows, as seen at the public server, differs depending on the source instance’s Availability Zone (AZ). All flows originating in AZ-a have NAT-GW-AZ-a’s elastic IP as the source, and all flows originating in AZ-b have NAT-GW-AZ-b’s elastic IP as the source.
Figure-2: NAT Gateway Deployment
Before this launch, if your deployment required over 55,000 simultaneous single-destination connections, then you would’ve needed multiple NAT Gateways and corresponding routing rules to achieve the desired scale. This meant separating your workloads into different subnets, and distributing traffic across multiple NAT Gateways.
To simplify your deployment, you can now assign up to eight Elastic IP addresses per NAT Gateway—an eight-fold increase in NAT Gateway’s scaling capabilities. Therefore, NAT Gateway now allows up to 440,000 simultaneous single-destination connections, as each associated IP address increases the limit linearly by 55,000 connections.
Using NAT Gateways with multiple Elastic IPs
Throughout this blog post, we use the sample network topology shown in Figure 3 below to highlight the new functionality of Amazon Web Services NAT Gateway. We’re using the distributed deployment for NAT Gateways, so our workloads and the NAT Gateway are in the same VPC. We show the steps to add multiple Amazon Web Services-assigned Elastic IP addresses to an existing NAT Gateway, but you can use this feature with BYOIP addresses, private IP addresses with the NAT Gateway, or a centralized NAT Gateway architecture. In this topology, the EC2 instances deployed in an autoscaling group constantly communicate with a test public server on TCP port 389. Traffic flows through the NAT Gateway and the IGW as shown in Figure 3 below.
Figure-3: Amazon VPC and NAT Gateway test deployment
Initially, the NAT Gateway we deployed has a single Elastic IP address associated with it. This means the NAT Gateway can support up to 55,000 simultaneous connections to the public server on TCP port 389. You will see port allocation errors on the NAT Gateway if the EC2 instances try to open over 55,000 connections to the unique destination represented by the public server receiving connections on TCP port 389. This is more likely to occur as the Auto Scaling group scales out and the number of instances increases. You can monitor the port allocation errors by using the ErrorPortAllocation
Let’s now associate multiple Elastic IPs with an existing NAT Gateway and some important considerations:
Step-1 (optional): Allocate an Elastic IPv4 address. This step is optional, and you can use an existing Elastic IP address as long as it isn’t associated with another resource. See the
Step-2: Navigate to your NAT Gateway and select the “Secondary IP addresses” tab. All secondary Elastic IPs, or BYOIPs, that you associate with the NAT Gateway are shown in Figure 4 below. Select “Edit secondary IPv4 address associations.”
Figure-4: Amazon Web Services NAT Gateway – Secondary IPv4 addresses tab
Step-3: Select the additional Elastic IP address that you’d like to associate with the NAT Gateway, and then select “Save changes.”
Figure-5: Select Secondary IP for NAT Gateway
Now the new Elastic IP address is ready, with an additional 55,000 simultaneous single-destination connections from the NAT Gateway. When you associate a secondary Elastic IP with a NAT Gateway, a private IPv4 address is automatically selected from the NAT Gateway’s subnet. This is shown in figure 6 below:
Figure 6: Secondary Elastic IP and private IPv4 addresses associated with NAT Gateway
The NAT Gateway uses a flow hash mechanism to select one associated IP address for a given flow. The fields included in this hash are the Elastic Network Interface ID of the source of traffic, the source and destination IP addresses, the source and destination ports, and the protocol. For TCP traffic, the TCP sequence number is also included in the hash calculation. Since the source and destination port numbers are included in the hashing algorithm, you may see different source IPs for applications like passive FTP that change the port numbers. For workloads that need outbound connections to these applications from a single source IP address, we recommend using NAT Gateways with a single associated Elastic IP address.
To test NAT Gateway’s flow hashing across multiple Elastic IPs, we used iPerf. Figure 7 below shows a snippet taken from the test public server’s console. You can see the NAT Gateway’s primary Elastic IP address (54.17x.yy.zz), and the secondary Elastic IP address (52.9.aa.bb).
Figure 7: Packet captures from the test server
Considerations
Consider the following when you associate multiple IP addresses with a NAT Gateway:
- Each secondary Elastic IP or BYOIP associated with an NAT Gateway automatically assigns an additional private IPv4 address to the NAT Gateway. Make sure that the NAT Gateway’s subnet has sufficient IP addresses available.
- If you’re using VPC Flow logs for the NAT Gateway’s Elastic Network Interface (ENI), then you’ll continue to see the NAT Gateway’s primary private IPv4 addresses. This is because a separate ENI isn’t created when you associate a secondary Elastic IP address with a NAT Gateway. You can use the pkt-srcaddr and pkt-dstaddr fields within VPC flow logs to identify the clients’ and server’s original IP addresses in the packet flow. See our documentation on
Logging IP traffic using VPC Flow Logs for details. - You can disassociate any secondary Elastic IPs associated with a NAT Gateway. At the time of disassociating a secondary IP from the NAT Gateway, you can provide the Connection Drain duration. This is the time the NAT Gateway waits before releasing the secondary IP. The default Connection Drain value is 350 seconds. See the
NAT Gateway documentation for details. - The NAT Gateway uses a flow hash to select one of the associated IP addresses for a given flow. The source and destination IP addresses of the flow are included in the hash. For applications, like passive FTP, that change port numbers during the lifetime of the flow, we recommend using NAT Gateways with a single Elastic IP address association.
- By default, you can associate two public IP addresses with a NAT Gateway. This is a soft limit, and you can get this quota increased by reaching out to
Amazon Web Services Support .
Conclusion
In this post, you learned how to associate/disassociate additional Elastic IP addresses with a NAT Gateway. This feature is available today, so try it out and let us know if you have questions about the features or this post by contacting
About the authors
Ankit Chadha
Ankit Chadha is a Networking Specialist Solutions Architect supporting Amazon Web Services Industries Accounts at Amazon Web Services. He has over 13 years of experience with designing and building various Networking solutions like MPLS backbones, overlay/underlay based data-centers and campus networks. In his spare time, Ankit enjoys playing cricket, earning his cat’s trust, and reading biographies.
Alexandra Huides
Alexandra Huides is a Principal Networking Specialist Solutions Architect within Strategic Accounts at Amazon Web Services. She focuses on helping customers build and develop network architectures for highly scalable and resilient Amazon Web Services environments. Alex is also a public speaker for Amazon Web Services, and she focuses on helping customers design and adopt highly scalable, resilient and flexible network architectures. Outside work, she loves sailing, especially catamarans, traveling, discovering new cultures, and reading
