Viewing Web Application Response Headers For Validating QIDs

Document created by John Delaroderie Employee on Aug 2, 2019
Version 1Show Document
  • View in full screen mode


Introduction

Whether you are manually validating QIDs for web application response headers or need to obtain an authorization token for header injection in your WAS scan, there are several ways you can extract them from your network traffic.  This document will cover several methods utilizing a variety of tools to best suit your needs and infrastructure. 


Response Headers QIDs

This manual review is applicable but not limited to the following QIDs:

 

11827   HTTP Security Header Not Detected
48001   Content-Security-Policy HTTP Security Header Not Detected
150028    Cookies Collected
150029    Session Cookies
150032    Session Cookie Does Not Contain The "secure" Attribute
150045    Session Cookie Does Not Contain The "HTTPOnly" Attribute
150060    HTTP Response Splitting
150081   X-Frame-Options Header is not set
150120   Session Cookie (Authentication Related) Does Not Contain The "secure" Attribute
150121   Session Cookie (Authentication Related) Does Not Contain The "HTTPOnly" Attribute
150122    Cookie Does Not Contain The "secure" Attribute
150123    Cookie Does Not Contain The "HTTPOnly" Attribute
150160   Session Cookie (Authentication Related) Set over Non-HTTPS Connection
150161   Session Cookie Does Not Contain the "Secure" Attribute
150135   HTTP Strict Transport Security (HSTS) header missing/misconfigured
150159   Session Cookie Set over Non-HTTPS Connection
150192   HTTP Response Header Injection
150202   Missing header: X-Content-Type-Options
150203   Misconfigured header: X-Content-Type-Options
150204   Missing header: X-XSS-Protection
150205   Misconfigured header: X-XSS-Protection
150206   Content-Security-Policy Not Implemented
150210    Information Disclosure via Response Header
226005   Invalid HTTP Header

 

For this example, we will take a look at CGI vulnerability QID 11827 and the related WAS QIDs: 150081, 150135, 150202, 150203, 150204, 150205.

 

Collectively, these QIDs evaluate a web application's response headers for the presence and correct configuration of X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, and Strict-Transport-Security

 

The Qualys KnowledgeBase defines these response headers as follows (taken from QID 11827):

X-Frame-Options: This HTTP response header improves the protection of web applications against clickjacking attacks. Clickjacking, also known as a "UI redress attack", allows an attacker to use multiple transparent or opaque layers to trick a targeted user into clicking on a button or link on another page when they were intending to click on the the top level page.

 

   Valid directives for X-Frame-Options are:
   X-Frame-Options: DENY - The page cannot be displayed in a frame, regardless of the site attempting to do so.
   X-Frame-Options: SAMEORIGIN - The page can only be displayed in a frame on the same origin as the page itself.
   X-Frame-Options: ALLOW-FROM RESOURCE-URL - The page can only be displayed in a frame on the specified origin.


X-XSS-Protection: This HTTP header enables the browser built-in Cross-Site Scripting (XSS) filter to prevent cross-site scripting attacks. X-XSS-Protection: 0; disables this functionality.

 

   Valid directives for X-XSS-Protections are:

   X-XSS-Protection: 1 - Enables XSS filtering (usually default in browsers). If a cross-site scripting attack is detected, the browser will sanitize the page (remove the unsafe parts).
   X-XSS-Protection: 1; mode=block - Enables XSS filtering. Rather than sanitizing the page, the browser will prevent rendering of the page if an attack is detected.
   X-XSS-Protection: 1; report=URI - Enables XSS filtering. If a cross-site scripting attack is detected, the browser will sanitize the page and report the violation. This uses the functionality of the CSP report-uri directive to send a report.
   X-XSS-Protection: 0 disables this directive and hence is also treated as not detected.


X-Content-Type-Options: This HTTP header prevents attacks based on MIME-type mismatch. The only possible value is nosniff. If your server returns X-Content-Type-Options: nosniff in the response, the browser will refuse to load the styles and scripts in case they have an incorrect MIME-type.

 

   A valid directive for X-Content-Type-Options: nosniff


Strict-Transport-Security: The HTTP Strict-Transport-Security response header (HSTS) is a security feature that lets a web site tell browsers that it should only be communicated with using HTTPS, instead of using HTTP.

 

   A valid HSTS directive Strict-Transport-Security: max-age=<expire-time>; [; includeSubDomains][; preload]


Response Headers and Redirects

When you interact with a web application, each request made by your web browser client will return a response code from the web application server.  While there are many different codes, the important ones we need to consider when reviewing web application server response headers are the 200 status codes and the 3XX status codes.

 

When a server returns a 200 status code, it is saying you your request was successfully received, understood, and accepted by the server. 

 

When a server returns a 3XX status code, the client must take additional steps to complete the request.  In other words, the request must be redirected to a different URL.  The most common 3XX responses are:

301 - Moved Permanently (the client should redirect to the URL in the "Location" header)

302 - Found (the client shoudl redirect to the URL in the "Location" header)

 

In all the examples below, we will look at responses returning a 200 status code.  We do not follow any redirects intentionally.  So what do you do when you receive a 3XX code and you are inspecting web application response headers?  If you are using one of the methods that utilizes a web client, your web client will automatically follow the redirect and you will see the new URL requested.  In this case you can ignore the 3XX responses and search for the first 200 response back from one of the redirection URLs.  If you are using a command line tool such as cURL or the provided Python script, no automatic redirection occurs.  In these instances, you would have to execute the cURL or Python script again with the redirection URL provided in the 3XX response. 

 

For example, using cURL (covered in Method 5 below) to navigate to the Qualys website using the URL qualys.com we see:

http://qualys.com redirects (301) to https://qualys.com

https://qualys.com redirects (302) to https://www.qualys.com

https://www.qualys.com returns a 200 response code and we can see the web application server response headers for this URL.

 

 

It does not matter if we try to connect to qualys.com, www.qualys.com, or http://www.qualys.com.  All requests return a 301 status code and tell us we need to instead use the URL https://www.qualys.com.  Once we use the new URL, we receive a 200 status code (Success!) and the server response headers are returned to us.

 

If we are using a client browser and capturing our traffic in a proxy intercept tool like OWASP ZAP (covered in method 4 below), we see the same redirection:

http://qualys.com redirects (301) to https://qualys.com

https://qualys.com redirects (302) to https://www.qualys.com

https://www.qualys.com returns a 200 response code and we can see the web application server response headers for this URL.

 

 

In any case, we simply keep following the redirections until we see the 200 response code indicating success.

 

Methods to View Response Headers

Now that we know what Qualys is looking for and how each header should be configured, let's take a look at how we can review the headers ourselves.  There are several methods available, and in this article we will demonstrate the following:

 

  1. Chrome Browser Developer Tools
  2. Firefox Browser Web Developer
  3. OWASP ZAP
  4. Command line with cURL
  5. Python Script

 

Of course, this list is incomplete - there are many other methods and tools to obtain web application response headers, and many client browsers have plugins available to assist you.  But the methods presented here will always work for you even if you are in an environment that blocks plugin installation.

 

Method 1: Chrome Browser Developer Tools

Chrome client browsers come preinstalled with Developer Tools.  To access Developer Tools, click on the 'more vertical' icon represented by 3 stacked dots in the upper right corner of your browser, select 'More Tools', and finally select 'Developer Tools'.

 

 

Once the Developer Tools have opened, select the 'Network' tab

 

 

Now you can browse to any site and capture the network traffic - including the web application server response headers.  In this example I will navigate to https://www.qualys.com.  You will see all each request made for source code, image files, style sheets, etc... as the web application loads.

 

 

The topmost request is for the URL www.qualys.com.  Note that the status code returned is 200 so we know that this request is not being redirected to a different URL.  If we click our mouse on this request, we will be presented with a new block containing the headers in both our request as well as in the server response.

 

 

Now we can review the actual response headers for X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, and Strict-Transport-Security.  We can see here that the server responded with the following:

 

Finally, we can review the valid directives for these response headers as outlined in the Qualys KnowledgeBase to verify if they are correctly set or not.

 

Method 2: Firefox Browser Web Developer

Firefox client browsers come preinstalled with Web Developer.  To access Web Developer, click on the 'Settingsl' icon represented by 3 stacked lines in the upper right corner of your browser, and select 'Web Developer'.

 

 

This action will open a new menu for the Web Developer. 

 

 

Select the 'Network' option. 

 

Now you can browse to any site and capture the network traffic - including the web application server response headers.  In this example I will navigate to https://www.qualys.com.  You will see all each request made for source code, image files, style sheets, etc... as the web application loads.

 

 

The topmost request is for the URL www.qualys.com.  Note that the status code returned is 200 so we know that this request is not being redirected to a different URL.If we click our mouse on this request, we will be presented with a new block containing the headers in both our request as well as in the server response.

 

 

Now we can review the actual response headers for X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, and Strict-Transport-Security.  We can see here that the server responded with the following:

 

Finally, we can review the valid directives for these response headers as outlined in the Qualys KnowledgeBase to verify if they are correctly set or not.

 

Method 3: OWASP ZAP

The Open Web Application Security Project (OWASP) Zed Attack Proxy (ZAP) is a free, opensource security tool that will allow you to quickly inspect web application server response headers.  It can be downloaded here: OWASP Zed Attack Proxy Project - OWASP

 

You will need to configure your client browser to use ZAP as a proxy.  There are various tutorials online that can assist you in getting it to run on your machine.  Once you have it set up and your client browser passing traffic through the proxy, you are ready to begin.

 

 

Now you can browse to any site from your client browser (Chrome, Firefox, etc) and capture the network traffic in ZAP - including the web application server response headers.  In this example I will navigate to https://www.qualys.com.  You will see all each request made for source code, image files, style sheets, etc... as the web application loads.  In order to do so, first navigate to the site in your client browser.

 

 

Once the web application has loaded, you can review the network traffic captured in ZAP.

 

 

The topmost request is for the URL www.qualys.com.  Note that the status code returned is 200 so we know that this request is not being redirected to a different URL.  If we click our mouse on this request, we will first be shown our request, and by selecting the 'Response' tab we can see the server response.

 

 

Now we can review the actual response headers for X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, and Strict-Transport-Security.  We can see here that the server responded with the following:

 

 

Finally, we can review the valid directives for these response headers as outlined in the Qualys KnowledgeBase to verify if they are correctly set or not.

 

Method 4: Command Line with cURL

cURL is a command line tool for transferring data to or from a web server over various different protocols, including HTTP.  Unlike the other methods demonstrated in this guide, it does not require you to browse to the site with a client web browser.  Instead, everything runs form a terminal with this useful command line tool.

 

The command is pretty straightforward and there are a lot of different options that can be appended to the command, but for this example we will restrict its use to just return the response headers.  However, you can find a complete listing of it's usage and various options here: curl Man Page - Linux - SS64.com 

 

Our command will be:

curl -s -I -k <URL>

or

curl --silent --head --insecure <URL>

The options in this command are:

-s or --silent: Silent or quiet mode. Don't show progress meter  or  error  messages.
-I or --head: Fetch the HTTP-header only
-k or --insecure: This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted in secure mode using the CA certificate  bundle  installed by  default. This  makes  all connections considered "insecure" fail unless -k, --insecure is used.

In this example, we will use https://www.qualys.com for the URL.

 

 

Once we hit enter, we are returned only the header from the web applications server..

 

 

Note that the status code returned is 200 (HTTP/2 200) so we know that this request is not being redirected to a different URL.  Now we can review the actual response headers for X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, and Strict-Transport-Security.  We can see here that the server responded with the following:

 

 

Finally, we can review the valid directives for these response headers as outlined in the Qualys KnowledgeBase to verify if they are correctly set or not.

 

Method 5: Python Script

 

As a final example, I want to demonstrate that you can also use scripts to extract the response headers from web application servers if you are attempting to automate your own security tools.  Here I am using Python, but really this could be performed by any number of languages.

 

Code:

 

# Sample Python script to display web application server response headers
#
# POC    : John Delaroderie - jdelaroderie@qualys.com
# Usage  : python show_headers.py <URL>
# Example: python show_headers.py https://www.qualys.com

import requests, sys, validators

# Our URL is passed in as a command line argument
url = sys.argv[1]

# Test for aa valid URL
if validators.url(url):

    # Make request - we are not following any redirects
    r = requests.get(url, allow_redirects=False)

    print ""
    print "HTTP Status Code:", r.status_code

    # Print out web application server response headers
    for key, value in r.headers.iteritems():
        print key, value

    print ""
# Otherwise we do not have a valid URL
else:
    # Provide exmample URL if URL is not properly formatted
    print "Please provide a valid URL.  e.g. https://www.qualys.com"

In this example, we will use https://www.qualys.com for the URL.

 

 

Once we hit enter, we are returned only the header from the web applications server..

 

 

Note that the status code returned is 200 so we know that this request is not being redirected to a different URL.  Now we can review the actual response headers for X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, and Strict-Transport-Security.  We can see here that the server responded with the following:

 

 

Finally, we can review the valid directives for these response headers as outlined in the Qualys KnowledgeBase to verify if they are correctly set or not.

 

Conclusion

In this article, we examined several methods for manually inspecting web application server response headers.   Now you have the tools to validate many QIDs yourself, and perhaps more importantly, ensure that any server response header related QIDs are remediated from your web applications. 

2 people found this helpful

Attachments

    Outcomes