NAT Hairpinning — 'Outside Works, Inside Fails'
The Call Comes In
- "External users can access our website fine — internal users cannot"
- "DNS resolves correctly, IP is correct, but connection times out from inside"
- "After we migrated to a new public IP, internal access broke"
- "Works from home VPN, fails from office"
- "Only internal users affected — same application, same server"
- "Developers cannot reach the API using the production URL from the office"
What Makes This Scenario Unique
Understanding the Hairpin Problem
When an internal user resolves a public domain name, they get the public IP address — say 203.0.113.10. They send traffic destined for 203.0.113.10. That traffic goes to the firewall's inside interface. The firewall looks up 203.0.113.10 in its routing table. The route points out the outside interface toward the internet.
At this point, the traffic needs to make a "U-turn" — arrive on the inside interface, get destination-NATted to the internal server IP (192.168.1.50), and then be forwarded back out the inside interface to reach the server. This U-turn is called NAT hairpinning or NAT reflection.
Without explicit hairpin configuration, this breaks in two specific ways on PAN-OS — and each has a different root cause.
What Most Engineers Try First (And Why It Fails)
- Adding DNS split-horizon (internal DNS returning internal IP) — this works but hides the actual problem and breaks applications that rely on the public URL for certificate matching
- Modifying security policy — policy is often not the primary issue; zone matching is
- Restarting the web server — server is fine, external users prove it
- Checking DNAT rule — rule exists and works for external users, so engineers assume it is correct for internal too
- Adding source NAT for internal users — added in the wrong place with wrong zones
The Two Failure Modes and Their Root Causes
Failure Mode 1: Security Policy Zone Mismatch
This is the most common failure. The DNAT rule translates 203.0.113.10 → 192.168.1.50. But after translation, the routing table sends the packet back out the inside interface — into the trust zone. The security policy was written as: untrust → dmz: allow port 443.
But the internal user traffic arrives from the trust zone, not the untrust zone. Policy lookup is: trust → trust (because server is in trust zone after routing). No rule matches. Traffic dropped.
The zone used in security policy evaluation is determined by INGRESS zone (where the packet came from) and EGRESS zone (where routing says it goes after NAT translation). For hairpin traffic: ingress is trust, egress is trust. Your existing rule says untrust to dmz. It never matches.
Failure Mode 2: Missing U-Turn Source NAT
Even when the security policy is correct, hairpin traffic fails because the source IP is not translated. Here is the sequence:
| Step | Source IP | Destination IP | Problem |
|---|---|---|---|
| Client sends | 192.168.1.100 | 203.0.113.10 | No problem yet |
| DNAT applied | 192.168.1.100 | 192.168.1.50 (after DNAT) | Source is still internal client |
| Packet reaches server | 192.168.1.100 | 192.168.1.50 | Server sees client IP directly |
| Server responds | 192.168.1.50 | 192.168.1.100 | Response goes DIRECTLY to client |
| Client receives | 192.168.1.50 | 192.168.1.100 | Client expected reply from 203.0.113.10 |
| TCP state mismatch | — | — | Client rejects packet — wrong source IP for this session |
The server responds directly to the client without going through the firewall — because they are on the same subnet. The client receives a packet from 192.168.1.50 but was expecting a response from 203.0.113.10. The TCP stack drops it. Connection fails.
The Actual TAC Debug Sequence
Step 1 — Confirm the Zone Mismatch via Policy Test
# Test from the internal client perspective # Source: internal client IP, Destination: PUBLIC IP of the server (pre-NAT) test security-policy-match from trust to untrust source 192.168.1.100 destination 203.0.113.10 protocol 6 destination-port 443 # If this returns the correct allow rule — policy is fine for pre-NAT match # Now test what happens AFTER routing (post-NAT zone) # Check where routing sends 203.0.113.10 traffic from inside test routing fib-lookup virtual-router default ip 203.0.113.10 # Then check where DNAT'd traffic (192.168.1.50) gets routed test routing fib-lookup virtual-router default ip 192.168.1.50
If fib-lookup for the translated IP (192.168.1.50) returns an interface in the trust zone, and your policy only allows untrust → dmz, the traffic is being dropped because the effective zone pair for the hairpin traffic is trust → trust.
Step 2 — Confirm with Session Browser
# Look for sessions from the internal client to the public IP show session all filter source 192.168.1.100 destination 203.0.113.10 # Also look for sessions after DNAT translation show session all filter source 192.168.1.100 destination 192.168.1.50 # If no sessions form at all — security policy is dropping before session creation # If sessions form but fail — likely the source NAT / U-turn issue # Check logs for denial reason show log traffic direction equal both | match "192.168.1.100.*203.0.113.10|203.0.113.10.*192.168.1.100"
Step 3 — Verify NAT Rule Ordering
# Show active NAT rules — order matters, first match wins show running nat-policy # Test which NAT rule matches for hairpin traffic # Source zone: trust (internal user), Destination: public IP test nat-policy-match from trust to untrust source 192.168.1.100 destination 203.0.113.10 protocol 6 destination-port 443
Pre-NAT vs Post-NAT in NAT Rule Matching
The Fix — Implementing U-Turn NAT
Hairpin NAT requires two NAT rules and a security policy that covers the correct zone pair. Here is the exact configuration logic:
NAT Rule 1 — Destination NAT (may already exist)
| Field | Value | Why |
|---|---|---|
| Source Zone | trust AND untrust | Must cover both internal and external users |
| Destination Zone | untrust | Pre-NAT destination zone — public IP routes out untrust |
| Destination Address | 203.0.113.10 | The public IP (pre-NAT destination) |
| Translated Destination | 192.168.1.50 | The internal server IP |
| Translated Port | 443 (if port forwarding) | Optional — only if port changes |
NAT Rule 2 — Source NAT for Hairpin (U-Turn)
| Field | Value | Why |
|---|---|---|
| Source Zone | trust | Only for internal users — external users must NOT hit this rule |
| Destination Zone | untrust | Pre-NAT destination zone |
| Destination Address | 203.0.113.10 | The public IP being accessed |
| Source Translation | Interface IP of inside interface (e.g., 192.168.1.1) | Makes server think firewall is the client — return goes through firewall |
| Type | Dynamic IP and Port (DIPP) | Standard source NAT type |
Why Source NAT Is the Key
Security Policy for Hairpin Zone Pair
| Field | Value |
|---|---|
| Source Zone | trust |
| Destination Zone | trust (or dmz — where the server actually is after DNAT routing) |
| Source Address | Internal subnets |
| Destination Address | 203.0.113.10 (pre-NAT public IP) OR 192.168.1.50 (post-NAT internal IP) |
| Application | web-browsing, ssl, or specific app |
| Action | allow |
Pre-NAT Address in Policy
Validation After Fix
# Test the full policy match for hairpin scenario test security-policy-match from trust to trust source 192.168.1.100 destination 203.0.113.10 protocol 6 destination-port 443 # Test NAT rule match for hairpin test nat-policy-match from trust to untrust source 192.168.1.100 destination 203.0.113.10 protocol 6 destination-port 443 # Have internal user attempt connection # Then confirm session forms with both NAT translations applied show session all filter source 192.168.1.100 destination 192.168.1.50 # Session should show: # address/port translation: source + destination # nat-rule: <your-hairpin-rule-name>
Permanent Solutions and Alternatives
Option 1: DNS Split-Horizon
Configure your internal DNS server to return the internal IP (192.168.1.50) for internal users querying the public domain name, while external DNS continues to return the public IP. Internal users connect directly to the internal IP, no hairpin needed.
Advantage: eliminates hairpin completely, reduces firewall load.
Disadvantage: DNS management overhead. If the internal server moves, two DNS records must be updated. Certificate SANs must match both internal and external FQDNs. Does not work for wildcard DNS.
Option 2: U-Turn NAT (as configured above)
Advantage: transparent to users — same URL works everywhere. No DNS complexity. Certificate matching works correctly.
Disadvantage: hairpin traffic passes through firewall twice, using two session entries per connection. At scale (hundreds of internal users), this can impact firewall session table capacity and throughput.