Purging: What, why, when, how, what happens to the data?

Document created by Martin Walker Employee on Jun 23, 2016Last modified by Martin Walker Employee on Apr 22, 2019
Version 2Show Document
  • View in full screen mode

What is “purging"?

Purging is one of a few maintenance activities that must be performed on a regular basis for most subscriptions.  


In this context “Purging” refers to the removal of stale asset data.  That is, data about assets that no longer exist in the environment.  In most environments host assets come and go on a regular basis.  In some highly ephemeral cloud environments especially, the host asset data in the subscription can rapidly become out of date. Purging can be performed via the UI or automated through API calls.


Why should we purge?

Quite simply, to keep the data in your subscription fresh and ensure that it most accurately reflects your environment.


When decomissioned stale assets are not purged the vulnerability findings and tickets previously identified or created for an IP address can never be closed.  Findings on stale assets will be included in security score calculations, will continue to show as unremediated and will skew things like remediation performance calculations, possibly showing you missing SLA targets, and may show on remediation and patch reports causing unnecessary work with remediation teams and undermining the trustworthiness of your VM data with those teams.  Additionally, if the IP address is reused for a new asset this may cause the findings from the old asset to become associated with new asset depending on type of scans being performed.  This can be particularly confusing if they are different types of assets for example Windows and Linux.  


When should we purge?

The frequency of purging depends on several factors including the size of the environment, how dynamic, how frequently IP addresses get re-used, level of automation, and how the data is being used.  As a rule of thumb for most organizations I would consider somewhere between monthly and quarterly purging.  This provides a reasonable balance between data accuracy and level of effort, and provides some leeway for missed scans due to reasons not related to decommissioning assets (e.g. network outage, host downtime, unplanned maintenance, other transient errors).  For highly automated environments purging can be automated via the API using whatever orchestration tools and mechanisms are being used to decommission assets.


How to purge?

The first step in purging is identifying the assets that need to be purged.  One mechanism is to use Assets->Asset Search and search for assets in your target tags or Asset Groups with a Last Found Date not within your time frame, e.g. 90 days.  You can use the Asset Search tool to create a new Asset Group containing these assets, apply a special tag to the assets, or even schedule or launch a scan against them to verify they are unreachable.  Assets can be purged directly at this point, or if they are placed into an Asset Group or tagged, then a report can be run for comparison against change management tickets or similar.


You could also achieve a similar thing by creating a tag based on this Asset Search criteria.  This tag could then be combined with other tags for your reporting purposes.  Bear in mind however that as tag rules are evaluated against an asset as scan results are processed, and as decommissioned assets obviously can’t be scanned as they no longer exist, the tag cannot be applied dynamically.  Instead you’ll need to edit the tag within Asset View, select “Re-evaluate rule on save” and save the tag.  Once Qualys has finished evaluating the tag in the back end it will be attached to each asset not scanned within the number of days selected.  The tag rule will look something like this:


<?xml version="1.0" encoding="UTF-8"?>








It is important however to close the loop on whatever process you are using.  For example, if purging is being driven by a report of decommissioned assets from a change management process, ensure the assets no longer exist before purging them by checking their last scan date or attempting a re-scan.  Similarly, if the first step in the purging process is to identify assets in Qualys that haven’t been scanned recently, make sure there aren’t any other reasons for not scanning them (e.g. network outage, host downtime, unplanned maintenance, other transient errors) and close the loop with change management, server teams etc first.


What happens to the data?

All findings data for the assets you are purging will be deleted from the host assets database.  This includes general information about the asset itself, all findings data, all remediation tickets associated with the asset, and any exceptions. 


It is important to note, the asset will not be removed from Host Assets, Asset Groups, or Authentication Records.  Additionally the IP still exists in schedule scans and scheduled reports.  The asset data and findings will not be removed from any existing raw scan results or reports that are saved. The IP will still exist in your subscription and may be counted against your license count (depending on the licensing model you use).  Importantly, if the IP is in the global Excluded Hosts list, it will remain there also, and in the IP list for any Blocked Ports in the Additional tab of any Option Profiles.


When purging assets you can select to purge VM data, PC data, or both.  Remember however, that once data is purged, no historical information is available and all associated tickets are closed and will not be included in trending calculations etc.


What happens if I remove an IP?

There is a slightly different option for cleaning up the asset data.  In Host Assets->New->Remove IP you have the option to remove addresses from your subscription.  This performs all the actions described above, but also removes the IP from your subscription (so it doesn’t count against license count).  The IP is not removed from Excluded IPs.  It is removed from Asset Groups and Scheduled Scans that target the IP directly (rather than using Asset Groups, ranges, or Tags).  Scheduled scans will be disabled at next scheduled run time. If the IP is contained in a range in an Asset Group it is not removed. The 'Removal' of an IP can take time, depending on how many data points it is included in.  We also have to process user's information to rebuild their 'User Scope' as security and visibility is handled very carefully.


What about agents?

IPs with an agent installed will remain on your host assets list. You can remove agents using the Cloud Agent module.  In order to purge Agent data you need to revoke or deactivate the agent from within the Cloud Agent module.


Purging OS Data Automatically During a Scan

There is an option in the scan Option Profile labeled "Purge old host data when OS is changed".  This will purge old host data, including QIDs, associated tickets, tags, Agentless Tracking GUID when we detect a major OS change.  This means Windows to Linux, Cisco to CheckPoint or something similar not simply a version upgrade such as Windows 2008 to 2012.  This does require a good OS report, one based on a TCP stack fingerprint e.g. "Windows 7/Windows 8/Windows 2012" or "Linux 2.3 / Linux 2.6 / F5" will not trigger a purge.  Asset ID does not change with this option.

8 people found this helpful