Using Palo Alto Networks Firewall Log Discovery

A Palo Alto Networks Firewall Log Discovery job finds the TLS servers on your internal network that clients connect to. The job gets this information from the decryption log of a Palo Alto Networks next-generation firewall.

When a client connects to a server over TLS, the firewall writes an entry in its decryption log. The entry shows the client, the server and the port. The job reads these entries and finds the sessions where the client and the server are both internal. From these sessions, the job makes a list of servers. Then the job scans only the servers on that list.

Because the list comes from real connections, the job can find servers that you did not know about. The job also does not scan addresses where there is no server.

The job scans the servers in the same way as a network discovery job. The Discovery engines connect directly to each server and read its certificate. The scan does not go through the firewall. Placement rules, exclusions, schedules, blackout times and results work the same as for network discovery jobs.

NOTE: This job is not the same as Onboard Discovery for Palo Alto Networks firewalls.

How the job works

Each time the job runs, it does these steps:

The job reads the part of the log that is new since the previous run. If the job has no log position, it reads back the number of hours in Log Lookback. For example, this happens on the first run.
The job makes a list of targets. A target is one server address and one port, for example 10.20.30.40:443. Each target is in the list only one time. The job does not add targets in excluded networks.
The Discovery engines scan the targets. Then they put the results in folders, as your placement rules specify.

Because each run reads only the new part of the log, a run does not scan a server again unless the server is in the new part of the log.

Limitations

One job reads one firewall. This job type does not support Panorama. To discover certificates through Panorama, use an onboard discovery job instead.
The job scans only IPv4 servers. It does not scan IPv6 servers.
The job uses only sessions between two internal hosts. It does not scan servers on the internet.
The job does not compare the certificates in the firewall log with your inventory.
The job does not make records for servers that it could not scan.
Retrieve Zones shows only zones on interfaces that have an IPv4 address. It does not show virtual wire, layer 2, tap or IPv6-only zones. For hosts in these zones, add their networks in Additional Internal Networks.

Prerequisites

Complete these prerequisites before you create the job.

On the firewall

Make a decryption policy rule for the traffic that you want to discover. A no-decrypt rule is sufficient. The job needs only the log entry. It does not need the decrypted traffic. If no decryption rule matches the traffic, the traffic is not in the decryption log.
Configure the rule to log successful TLS handshakes. By default, decryption rules log only unsuccessful handshakes.

IMPORTANT: When you log successful handshakes, the decryption log can become as large as the Traffic log. By default, the firewall uses only 1% of its log storage for the decryption log. Before you enable this logging, examine the log storage quotas on the firewall.
Make a firewall account that can use the XML API. The job only reads data from the firewall. It does not change the firewall. A read-only administrator account is sufficient. If you use a custom admin role, give the role these XML API permissions: Log and Operational Requests.

Network access

These computers must be able to connect to the management interface of the firewall with HTTPS (port 443 by default):

All Certificate Manager - Self-Hosted servers that run Discovery.
The web server. The web server connects to the firewall when you click Retrieve Zones.

The Discovery engines must also be able to connect to the discovered servers on their ports. This is the same as for network discovery.

In Certificate Manager - Self-Hosted

A device object for the firewall. The device object must have the host name or IP address of the firewall. It must also have a credential for the firewall account.
Permission to create discovery jobs. If you use Credential Override, you also need Read permission on that credential.
Recommended: Set Certificate Verification to Basic or Strict, as described in the next section.

Securing the connection to the firewall

Certificate Manager - Self-Hosted examines the HTTPS certificate of the firewall only when Certificate Verification is Basic or Strict.

CAUTION: The default value is None. With None, Certificate Manager - Self-Hosted accepts all certificates. Then an attacker can pretend to be the firewall and get the password of the firewall account.

When the value is Basic or Strict, the management certificate of the firewall must meet these requirements:

The certificate must be issued by a CA that the Discovery servers and the web server trust.
The name in the certificate must match the host name or IP address on the device object.
Strict only: the certificate must include the Server Authentication extended key usage.

By default, the management certificate of the firewall is self-signed. A self-signed certificate does not meet these requirements unless your servers trust it.

The Certificate Verification setting applies to all outgoing HTTPS connections from the server, not only to this job.

Creating a Palo Alto Networks Firewall Log Discovery job

IMPORTANT: Before you start, complete the prerequisites.

To create a Palo Alto Networks Firewall Log Discovery job

On the Certificate Manager - Self-Hosted menu bar, click Configuration > Jobs.
Click + Create New Job. Select Palo Alto Networks Firewall Log Discovery, and then click Start.
In the Job Details section, enter a Name and the Contacts for the job. You can also enter a Description.
In the Firewall section, select the Firewall Device. If necessary, enter values for Credential Override, XML API Port and Virtual System.
In the Internal Networks section, specify the internal traffic. You must enter at least one zone or one network.
1. Click Retrieve Zones. Then, in Internal Zones, select the zones of your internal network.
2. (Optional) In Additional Internal Networks, enter internal networks that are not in an internal zone.
3. (Optional) In Excluded Networks, enter the ranges that the job must never scan.
(Optional) In the Exclusions section, select discovery exclusion lists.
(Optional) In the Limits section, change the limits for each run. To use the default value, leave the field empty.
In the Placement Rules section, do one or both of these steps:
- To add an existing rule, use Select rule to add.
- To make a new rule, click + Create New Rule.
In If no rules apply, put results into this folder, select a folder.
Click Create.

The job does not run on a schedule until you add a schedule.

Job Details settings

Name: Required. The name of the job. The name shows in the Jobs list.
Description: Optional. A description of the job. For example, enter the name of the firewall that the job reads.
Contacts: Required. The people who are responsible for the job.

Firewall settings

Firewall Device: Required. The firewall that the job reads. The job uses the host name and the credential from the device object. The job does not use the port setting of the device object. To set the port, use XML API Port.
Credential Override: Optional. A different credential for this job only. For example, use a read-only account. Supported credential types: Username Password, CyberArk Username Password and Adaptable. You must have Read permission on the credential. Without this permission, you cannot save the job.
XML API Port: Optional. The XML API port of the firewall. If the field is empty, the job uses port 443. Values: 1–65535.
Virtual System: Optional. The virtual system (vsys) that has the logs. If the field is empty, the job uses the default virtual system of the firewall. You can use letters, numbers, periods (.), hyphens (-) and underscores (_).

NOTE: If you change Firewall Device, Credential Override or Virtual System on an existing job, the next run starts again at the log lookback.

Internal Networks settings

The job uses a session only when the client and the server are both internal. You must enter at least one internal zone or one internal network. The job scans only IPv4 servers.

Internal Zones: The firewall zones of your internal network. To get the list of zones from the firewall, click Retrieve Zones. To use Retrieve Zones, you need Write permission on the job. For a new job, you need permission to create discovery jobs.
Additional Internal Networks: Internal networks that are not in an internal zone. Also use this field for internal networks that you connect to through NAT. Enter one network on each line. Use CIDR format, for example 10.20.0.0/16.
Excluded Networks: Optional. Networks that the job never scans, even when the network is in an internal zone. Enter one network on each line. Use CIDR format, for example 10.20.30.0/24. To exclude a full zone, do not add the zone to Internal Zones. TIP: If the firewall uses destination NAT, add your public address ranges here. If you do not add them, the job can scan public addresses.
Networks Reported by the Firewall: Read-only. This table shows the networks of each zone and interface. The table shows only networks that are directly connected to the firewall. Networks behind other routers are not in the table, but their zones still include them. The job does not scan the networks in this table, and it does not add them to its settings. Before the job connects to the firewall, the table shows "No networks have been retrieved from the firewall yet."

Exclusions settings

Exclusions: Discovery exclusion lists. Use them for targets that the job must not scan, and for results that the job must ignore. Network discovery jobs use the same lists.

Limits settings

Log Lookback (hours): The number of hours of log that the first run reads. The job reads this period again after Reset Log Position. The job also reads this period again if you change the firewall, the virtual system or the credential override. If the field is empty, the value is 26 hours. Values: 1–4380 (six months).
Maximum Log Entries per Run: The maximum number of log entries that one run reads. When a run reads this number of entries, the run stops. The next run continues from that point. If this occurs frequently, the job reads the log more slowly than the firewall writes it. If the field is empty, the value is 250,000. Values: 5,000–1,000,000.
Scan Addresses Between Targets: When selected, the job scans all addresses between targets that use the same port and are in the same /24 network. This makes fewer scan jobs, so scans can be faster. But the job also scans addresses that are not in the firewall log. The job never scans excluded networks. Default: not selected.

Placement Rules settings

Placement rules: The rules that put discovered certificates and devices into folders. To make a new rule, click + Create New Rule. To add an existing rule, use Select rule to add.

If no rules apply, put results into this folder: Required. The folder for results that do not match a placement rule.

About the log position

The log position is the point in the firewall log where the last run stopped. The job saves the log position after each run. The next run starts at the log position, even if the previous run was a long time ago.

The first run starts at the log lookback. The first run after a reset also starts at the log lookback.
If you change Firewall Device, Credential Override or Virtual System, the next run starts again at the log lookback.
If you change the host name or the account on the device object, the job also starts again at the log lookback. This occurs at the next run. Before the next run, the job page continues to show the old log position. If you change only the password, the log position does not change.
If you change zones, networks, exclusions or limits, the log position does not change. The new settings apply only to log entries that the job reads after the change. To apply the new settings to older log entries, reset the log position.
If you delete a discovered certificate or device, the job does not find it again. The job finds it again only if the server is in a new part of the log.

IMPORTANT: If runs frequently stop at Maximum Log Entries per Run, the job reads the log more slowly than the firewall writes it. The firewall can delete old log entries before the job reads them. Then the job does not find those servers. To prevent this, increase Maximum Log Entries per Run, or run the job more frequently.

Resetting the log position

Reset the log position when you want the next run to read the log lookback period again. For example, reset it after you change the internal zones or networks.

IMPORTANT: You need Write permission on the job. Reset Log Position is available only when all of these conditions are true:

The job is enabled.
The job is not running.
The job has run at least one time.

To reset the log position

Open the job, or find the job in the Jobs list.
Click Reset Log Position. On the job page, this button is next to Run Now. In the Jobs list, it is in the actions for the job row.
In the Reset the log position? dialog box, click Yes, Reset.

The next run starts at the log lookback. It scans the servers that it finds in that part of the log. Servers from earlier runs are scanned again only if they are in that part of the log. The system logs the reset with your user name.

Viewing job results

The Last Run section of the job shows the results of the most recent run.

Status: The status of the job. If a run did not scan any servers, the status gives the reason.
Log Entries Read: The number of log entries that the run read.
Targets: The number of targets that the run sent for scanning. A target is one server address and one port.
Unique Certificate Fingerprints: The number of different server certificates in the firewall log. Use this number for information only. The number can be smaller than the number of certificates that the scans find, because the firewall cannot see the certificate in a TLS 1.3 session that it does not decrypt.
Log Read Time: The time, in seconds, that the run used to read the log.
Next Run Resumes From: The point where the next run starts to read the log, in the local time of the firewall. If there is no log position, the field shows "The next run starts at the log lookback."

If a run stopped early, a warning shows above these fields. The warning gives the reason and the action to take. A run can stop early for these reasons:

The run read the maximum number of log entries.
The job could not read a page of the log.
The firewall did not answer in time, so the job read a shorter time period.

Discovered certificates and devices show in the inventory and in the job results. This is the same as for network discovery.

Status messages and suggested action

No decrypted sessions between known internal hosts were found in the firewall log: Make sure that a decryption rule covers the traffic and logs successful handshakes. Make sure that your internal zones and networks are correct.
No internal zones or networks are configured, so no traffic can be identified as internal: Add at least one internal zone or one internal network.
The firewall could not be reached. Error: …: Make sure that the servers can connect to the firewall and that the firewall account is correct. The error message gives more information.
The firewall answered, but its clock could not be read, so the firewall log was not read: Contact Support.
The firewall's decryption log could not be read. Error: …: Make sure that the firewall account has permission to read logs.
The run was stopped before the firewall log was read. The next run reads the same period again: No action is necessary.
The credential store could not be opened, so the firewall's credential could not be read. Result: …: Make sure that the server can access its credential store.
The discovery is missing required configuration: …: Correct the setting that the message identifies. For example, the device object has no host name.

Events for notification rules

Use these events to configure notifications for this job.

Discovery - Firewall Unreachable: The job could not connect to the firewall, or could not log in.
Discovery - Firewall Log Query Failed: The firewall sent an error when the job read the log.
Discovery - Firewall Log Entries Truncated: A run stopped at Maximum Log Entries per Run. If this occurs frequently, the job reads the log too slowly.
Discovery - Firewall Log Window Narrowed: The firewall answered too slowly, so the run read a shorter part of the log. Later runs read the remaining part.
Discovery - Firewall Discovery No Internal Networks: The job has no internal zones or networks.
Discovery - Firewall Discovery Not Configured: A required setting is missing or is not valid.
Discovery - Firewall Zones Requested: A user clicked Retrieve Zones.
Discovery - Firewall Log Position Reset: A user reset the log position. The event includes the user name.

Technical Insights

This section gives more details about how the job works. Use it for troubleshooting or integration.

Run sequence

The job gets an API key with the device credential or the override credential. Then the job runs show clock to get the local time of the firewall. The log uses this local time.
The job runs show interface all and saves the directly connected networks. These networks show in Networks Reported by the Firewall. If this step fails, the run continues and logs a warning.
The job selects the time period to read. The period starts at the log position or at the log lookback. The period ends two minutes before the current time of the firewall.
The job reads the decryption log in pages. Each page has a maximum of 5,000 entries. The job stops when it reaches Maximum Log Entries per Run. The query asks the firewall for internal sessions only. In an internal session, the source and the destination are both in an internal zone or an additional internal network.
The job removes sessions that have an end in an excluded network. The job also removes destinations that it cannot scan: IPv6, 0.0.0.0/8, 127.0.0.0/8, 169.254.0.0/16, 224.0.0.0/4 and broadcast addresses. If a log entry has no port, the job uses port 443.
The job makes a list of different host:port targets. The job sends the list to the network discovery scan engine.
After the job sends the targets, it saves the run results and the new log position. If the run read the full period, the log position is the end of the period. If the run stopped early, the log position is the last entry that the run read.

Edge cases

Firewall timeout: If the first page does not arrive in time, the job asks for one quarter of the time period. The job does this again until the firewall answers, or until the period is one hour or less. The run records that it stopped early. Later runs read the remaining part of the log.
Run stopped during reading: The job does not save anything. The next run reads the same period again.
No targets: The run ends with a status message. The log position still moves forward.
Run results not saved (event 00070074): The next run reads the same period again.
Host or account change on the device object: The job finds the change at the start of the next run. Then it starts again at the log lookback.
End of daylight saving time: The firewall clock moves back one hour. The next run reads one hour of the log again. It does not read the full lookback period again.
Retrieve Zones: The web server connects to the firewall, not the Discovery engines.

Firewall API usage

The job only reads data. It never changes the firewall configuration. The job uses these calls: API key generation, show clock, show interface all and decryption log queries. A custom admin role with the XML API permissions Log and Operational Requests is probably sufficient for these calls.

Destination NAT

PAN-OS logs the destination address before NAT. Because of this, a session through destination NAT can add a public address to the targets. To prevent this, add the public address range to Excluded Networks.

Permission model

Scheduled runs and Run Now use the permissions of the Discovery service. They do not use the permissions of the user.
The user does not need Read permission on the credential of the device object. To save a job that has a Credential Override, the user needs Read permission on that credential.
Retrieve Zones and Reset Log Position need Write permission on the job. For Retrieve Zones on a new job, permission to create discovery jobs is sufficient.

Full event reference

All event names start with "Discovery - ".

ID	Name	Severity	Logged when
0007006A	Firewall Targets Not IPv4	Info	The job skipped destinations that are not IPv4.
0007006B	Firewall Log Entries Truncated	Warning	The run stopped at the maximum number of log entries.
0007006C	Firewall Log Window Narrowed	Warning	The first page did not arrive in time, so the job read a shorter period.
0007006D	Firewall Log Processing Started	Info	The run started to read the log.
0007006E	Firewall Log Processing Completed	Info	The run finished reading the log.
0007006F	Firewall Clock Unreadable	Error	The job could not read the output of `show clock`.
00070063	Firewall Networks Refreshed	Debug	The job saved the directly connected networks.
00070064	Firewall Network Refresh Failed	Warning	The job could not read the directly connected networks. The run continues.
00070065	Firewall Log Query Failed	Error	The firewall sent an error for a log page.
00070066	Firewall Targets Resolved	Info	The job made targets from the log entries.
00070067	Firewall Discovery No Internal Networks	Error	No internal zone or network is configured.
00070068	Firewall Discovery Not Configured	Error	A required setting is missing or is not valid.
00070069	Firewall Unreachable	Error	The job could not connect to the firewall, or could not log in.
00070070	Firewall Log Job Id Missing	Error	The firewall accepted the log query but did not send a job ID.
00070071	Firewall Log Response Unreadable	Error	The job could not read a log page.
00070072	Firewall Log Job Timed Out	Error	A log query did not finish in approximately one minute.
00070073	Firewall Networks Not Saved	Warning	The job could not save the directly connected networks.
00070074	Firewall Run Summary Not Saved	Error	The job did not save the run results and the log position. The next run reads the same period again.
00070075	Firewall Zones Requested	Info	A user clicked Retrieve Zones.
00070076	Firewall Log Position Reset	Info	A user clicked Reset Log Position.