John Malon

Identifying and Removing Duplicate Unique IDs (UUIDs) in Qualys (Revisited)

Discussion created by John Malon on Mar 24, 2020

Identifying and Removing Duplicate Unique IDs (UUIDs) in Qualys

 

Duplicate UUIDs normally appear when an asset is “cloned” that already contains a Qualys UUID. The UUID is stored in the registry key HKLM\SOFTWARE\Qualys\HostID on Windows systems and in the /etc/qualys/hostid file on Linux servers. Cloning is a great time saver, so just remove the UUID before or after the cloning process to prevent duplicate UUIDs.

 

Signs there a duplicate UUIDs in your network
1. If you scan a number of assets in your environment and you can see they were successfully scanned in the raw scan report, but when you try to find the asset in asset search, the asset cannot be found. This happens because Qualys combines the raw scan results, so two assets with the same UUID become one asset, thus you can find one but not the other.

 

2. You have a number of unused authentication records that no matter what you do they don’t get used. You may also see a flip flopping of unused authentication records, i.e. you scan asset IP 1.2.3.4 and the authentication record for 1.2.3.5 becomes unused. If you scan IP 1.2.3.5 the authentication record for IP 1.2.3.4 becomes unused.

 

Ways to find duplicate UUIDs in your network – small number of IPs
1. View a raw Qualys scan report and copy the “Successfully Scanned Hosts” IP address list from within the
appendix section, i.e. 1.2.3.1, 1.2.3.2, 1.2.3.3, 1.2.3.4, 1.2.4.5 and paste it into the asset search IP ranges box and
press the search button. If for example 1.2.3.3 does not appear in the search results, there’s a good chance it has a duplicate UUID, assuming fully authenticated scans are working properly.


With a small number of IPs like this example you can remote into 1.2.3.3 and obtain the UUID from the registry setting HKLM\SOFTWARE\Qualys\HostID or the file /etc/qualys/hostid and record it. Then perform an asset search for all asset groups searching for QID 45179 with matching results for the host ID you obtained, i.e. 11111111-2222-3333-4444-555555555555.

 

Asset Search for QID 45179


If another asset is returned, i.e. 1.2.3.1, with that host ID you’ve found your duplicate ID. Remote back into
1.2.3.3 and remove the hostid. Because all of the vulnerabilities of 1.2.3.1 and 1.2.3.3 have been combined into
1.2.3.1, you’ll want to purge asset 1.2.3.1 from Qualys and rescan both 1.2.3.1 and 1.2.3.3 as soon as possible.
When scanning has completed, do an asset search to make sure both 1.2.3.1 and 1.2.3.3 are found and their
authentication records have been used.

 

Ways to find duplicate UUIDs in your network – large number of IPs
1. If you have a tool, i.e. BigFix, you may be able to create a report that will tell you the UUID of each asset. You can then export the report as a CSV, import it into Excel, and highlight assets with the same UUID.

 

2. Without such a tool, you can find the information using Qualys, but it is a lot of work.

 

a. Download the raw scan reports for each of your scan groups, i.e. main location, remote offices, and disaster recovery site, as CSV files.
b. Place each of the CSV files into the same empty folder, i.e. c:\Qualys.
c. Combine the reports into one CSV file by going to the Windows command prompt, changing to the appropriate directory, i.e. c:\Qualys, and typing copy *.csv combined.csv.
d. Open the combined.csv file in Excel. If the file is too large, the 32-bit version of Excel may not be able to open it, so you may need to work with each individual CSV file.
e. Rename the file to something more meaningful, i.e. Qualys – Combined Assets – mm/dd/yyyy.csv
f. Save it in the XLSX format. This step will reduce the file size and Excel seems to perform better.
g. Remove unnecessary header information and columns. Be sure to keep columns IP, DNS, NetBIOS, OS, QID, and Results. Also remove any odd IP addresses at the bottom of the sheet. This step will also reduce the file size and allow Excel to perform better.
h. Save the XLSX file before proceeding.
i. Filter the sheet by QID 45179.
j. Ctrl + A to select all and then cut and paste the values into a new Excel tab/sheet.
k. On the new tab, sort the sheet by the results column.
l. Highlight the results column and go to Home > Conditional Formatting > Highlight Cells Rules > Duplicate Values… >

 

Excel - Format Duplicate Cells
m. If any duplicates are found, go to Data > Sort >

 

Excel - Sort Duplicates

n. The results will look something like this.

 

Excel - Example of Duplicate UUIDs

 

o. Purge the duplicate assets from Qualys.
p. Remove the UUIDs from the assets in either the registry at HKLM\SOFTWARE\Qualys\HostID or the /etc/qualys/hostid file.
q. Rescan the affected assets at the earliest opportunity.
r. When scanning has completed, do an asset search to make sure all of the rescanned assets are found and their authentication records have been used.

Outcomes