TAC Debugging Toolkit — Real Sequences, Not Command Lists

Why Command Lists Are Not Enough

Most PAN-OS documentation and training courses give you lists of commands. What they do not give you is the decision logic — when to run which command, what output to look for, what to do when you find it, and when to move to the next tool.

This module covers the TAC debugging toolkit as sequences with decision points — the actual thought process that separates a 10-minute fix from a 3-hour troubleshooting session.

The Packet Diagnostic Framework — packet-diag

Packet-diag is the most powerful TAC debugging tool but also the most commonly misused. It has two modes and most engineers only use one.

Mode 1: Log Mode (What Most Engineers Use)

pan-os-cli

# Set up the filter for affected traffic
debug dataplane packet-diag set filter match source 10.1.1.100
debug dataplane packet-diag set filter match destination 203.0.113.10

# Enable log output
debug dataplane packet-diag set log on

# Run for 30-60 seconds while user reproduces issue

# View the log
show debug dataplane packet-diag log

# Interpret the output:
# "flow_basic" lines show packet traversal through processing stages
# "session_install" means a new session was created
# "nat" lines show translation being applied
# "forward" means packet is being forwarded out
# "drop" with a reason code = where the problem is

# Clear when done (ALWAYS clear — leaving filter active impacts performance)
debug dataplane packet-diag clear filter
debug dataplane packet-diag clear log

Mode 2: Capture Mode (TAC Uses This for Hard Problems)

pan-os-cli

# Capture at multiple stages simultaneously
# Stages: ingress, firewall (after policy), egress, drop

# Set filter
debug dataplane packet-diag set filter match source 10.1.1.100

# Enable capture at all stages
debug dataplane packet-diag set capture stage ingress file /tmp/cap-ingress.pcap
debug dataplane packet-diag set capture stage firewall file /tmp/cap-fw.pcap
debug dataplane packet-diag set capture stage egress file /tmp/cap-egress.pcap
debug dataplane packet-diag set capture stage drop file /tmp/cap-drop.pcap

# Start capture
debug dataplane packet-diag set capture on

# Wait for user to reproduce (30-60 seconds)

# Stop capture
debug dataplane packet-diag set capture off

# Export captures for Wireshark analysis
scp export pcap from mgmt-pcap to <destination>

# Or view summary on device
show debug dataplane packet-diag capture brief

Reading Stage Captures Like TAC

If a packet appears in the ingress capture but not in the firewall capture — it was dropped before policy evaluation (zone protection, DoS policy, or interface issue). If it appears in the firewall capture but not in egress — security policy dropped it. If it appears in egress but the user reports no response — the packet left the firewall but the path onward is broken, or the return is not coming back. If it appears only in the drop capture — check the drop reason field.

Flow Basic — The TAC First-Look Tool

Flow basic gives a real-time view of packet processing decisions without a filter. It is noisier than packet-diag but faster to set up and excellent for confirming whether a specific type of traffic is even arriving at the firewall.

pan-os-cli

# Enable flow basic logging
debug dataplane internal forwarding-table lookup <destination-ip>

# This tells you immediately what the firewall's forwarding table says
# about where to send traffic for a specific destination — useful for
# confirming routing before running a full packet capture

# For session-level flow tracing:
# Not directly available without packet-diag, but global counters
# combined with session table give equivalent information

# Quick session check — is traffic forming sessions at all?
show session all filter source <client-ip> | match count

# If count = 0: traffic not reaching firewall or being dropped pre-session
# If count > 0: sessions forming, issue is within the session lifecycle

Test Commands — Validate Before You Change

Test commands simulate what PAN-OS would do for a specific packet without actually sending traffic. TAC uses these to validate the impact of a configuration change before committing it — and to diagnose what policy is matching without needing live traffic.

Security Policy Testing

pan-os-cli

# Test which security rule matches for specific traffic
test security-policy-match   from <source-zone>   to <destination-zone>   source <source-ip>   destination <destination-ip>   application <app-id>   destination-port <port>   protocol 6

# Examples:
test security-policy-match from trust to untrust   source 10.1.1.100   destination 8.8.8.8   application dns   destination-port 53   protocol 17

# Test with specific username (if User-ID is configured)
test security-policy-match from trust to untrust   source 10.1.1.100   destination 203.0.113.10   application ssl   destination-port 443   protocol 6   source-user domain\john.doe

Routing and NAT Testing

pan-os-cli

# Validate routing for a specific destination
test routing fib-lookup virtual-router default ip 203.0.113.10

# Test NAT rule matching
test nat-policy-match   from trust   to untrust   source 10.1.1.100   destination 203.0.113.10   protocol 6   destination-port 443

# Test PBF matching
test pbf-rule-match   from trust   source 10.1.1.100   destination 203.0.113.10   protocol 6   port 443

# Test decryption policy matching
test decryption-policy-match   from trust   to untrust   source 10.1.1.100   destination 203.0.113.10   destination-port 443

Authentication Testing

pan-os-cli

# Test authentication for a specific user against a profile
test authentication authentication-profile <profile-name>   username <username>   password

# Test RADIUS server reachability
test authentication authentication-profile <radius-profile>   username testuser   password testpassword

# Validate that User-ID is mapping correctly
show user ip-user-mapping ip <ip-address>

# Check full user mapping table
show user ip-user-mapping all

Log Correlation — Reading the Full Story

No single log type tells the complete story. TAC always correlates across multiple log types for the same time window and same 5-tuple.

Log Type	What to Look For	CLI Command
Traffic	Action (allow/deny), App-ID, rule name, bytes	show log traffic direction equal forward
Threat	IPS blocks, URL filtering, file blocking decisions	show log threat direction equal forward
Decryption	SSL errors, excluded traffic, cipher details	show log decryption direction equal forward
Authentication	Auth success/failure, method used, user identity	show log authentication direction equal forward
System	Firewall service events, HA state changes, content updates	show log system
Config	Configuration changes with timestamp and admin name	show log config

pan-os-cli

# Correlating logs for a specific incident time window
# Replace with actual timestamp range

show log traffic start-time equal 2026/05/26/08:00:00   end-time equal 2026/05/26/08:30:00   source equal 10.1.1.100

# Cross-reference with threat log for same time/source
show log threat start-time equal 2026/05/26/08:00:00   end-time equal 2026/05/26/08:30:00   source equal 10.1.1.100

# Check config log — did something change around the outage time?
show log config start-time equal 2026/05/26/07:00:00   end-time equal 2026/05/26/09:00:00

Global Counters — The Health Dashboard

Global counters are the fastest performance and health view. Here are the counters that matter most during an outage and what each tells you.

pan-os-cli

# Full drop counter view with delta (reset between samples)
show counter global filter delta yes severity drop

# Key counters by category:

# ROUTING ISSUES:
# flow_fwd_l3_noroute       — no route to destination, packet dropped
# flow_fwd_l3_loopback_err  — routing loop detected

# POLICY ISSUES:
# flow_policy_deny          — security policy deny action
# flow_policy_nat_deny      — post-NAT security policy deny

# NAT ISSUES:
# flow_nat_no_translation   — no NAT rule matched
# flow_nat_pool_exhausted   — SNAT pool has no available IPs

# TCP STATE ISSUES:
# flow_tcp_rst_from_client  — client sent RST
# flow_tcp_rst_from_server  — server sent RST (often asymmetric routing)
# flow_tcp_out_of_window    — packet outside TCP window (common in asymmetric routing)

# SSL/DECRYPTION ISSUES:
# decrypt_error             — SSL decryption failure
# ssl_cert_error            — certificate validation error

# RESOURCE ISSUES:
# session_max               — session table full (critical)
# flow_fwd_dst_nat_pool_exhausted — DNAT translation pool exhausted

session_max Is a Critical Alert

If you see session_max incrementing during an outage, the firewall has hit its maximum session table capacity. New connections are being rejected system-wide, not because of policy but because there is no space for new sessions. Immediate actions: identify and clear unused sessions from long-running idle connections, investigate whether a SYN flood or session table attack is occurring, and check whether an application is creating excessive sessions.

The TAC Escalation Package

When an issue cannot be resolved with the above toolkit and must be escalated to vendor TAC, collect this information before opening the case. It eliminates back-and-forth and gets to resolution faster.

pan-os-cli

# 1. System information
show system info > /tmp/sysinfo.txt

# 2. Capture at all stages during reproduction
debug dataplane packet-diag set filter match source <client-ip>
debug dataplane packet-diag set capture stage ingress file /tmp/esc-ingress.pcap
debug dataplane packet-diag set capture stage drop file /tmp/esc-drop.pcap
debug dataplane packet-diag set capture on
# Reproduce issue
debug dataplane packet-diag set capture off

# 3. Session state at time of failure
show session all filter source <client-ip> > /tmp/sessions.txt

# 4. Relevant log entries (30-minute window around issue)
show log traffic direction equal forward   | match "<affected-ip>" > /tmp/traffic-logs.txt

# 5. Global counters at peak of issue
show counter global filter delta yes severity drop > /tmp/counters.txt

# 6. Tech support file (TAC will always ask for this)
request tech-support
# Note the file location from output, then SCP it out

Previous Module Next Module