Modifying Headers In HTTP(s) Requests In UI Automation Testing

About The Author

Nafees is an IT student who is keen to learn new things. His motto is “always be ready to learn, unlearn and relearn.”

More about
Nafees ↬

There are various methods to modify headers. You can modify headers by using browser extensions or proxy apps (such as Charles and Proxyman) that intercept the request and let you modify the response by including the headers.

But first, let’s start at the beginning.

What Are HTTP(s) Headers?

HTTP(s) Headers are key-value pairs that can be used by the client or server to pass additional information along with an HTTP(s) request or response. They hold additional information about the data being sent. An HTTP(s) header consists of a case-insensitive name followed by a colon (:), then by its value. Field names are case insensitive but field values are case sensitive. A header can have multiple values which are separated by commas.

Modifying Headers: How Can This Be Helpful?

I was working on an application that opened the provided webpage and then give the user an option to modify elements, add events, add analytics, and so on. without any need to code. This was being done by loading the page in iframe and giving various options to the user on top of it. I tried with loads of websites to see how they behave in iframe. I observed that most of the websites don’t work in iframe due to x-frame-options and content-security-policy headers.

Almost all websites have a header X-frame-options set to deny or sameorigin due to which the browser does not allow to load the webpage in an iframe or doesn’t when any cross-origin request tries to load it in an iframe. Also, the content-security-policy header has frame-ancestors directive which prevents this.

It was very important to load the page in iframe, I was wandering around the internet to find a way to load it. It was evident that I needed to override the X-frame-options header to allowall or remove it altogether. That was when I stumbled upon Requestly extension which gave me the feature to modify the X-frame-options header by matching the page URL and hence allowing me to override the X-frame-options header when debugging.

That was when I first got to witness the power of network headers. They carry data about the data being transferred. The ability to modify the headers of traffic that pass through your browser is a great tool to have. Besides overriding X-frame-options, you can delete headers to minimize online tracking, override content-security-policy header, test sites in production, and so on.

While testing web applications, modifying headers provides a great hack:

When I got to know about the automated testing of web apps, it occurred to me that modifying headers should be a feature there due to its immense applicability in web app testing. Therefore I decided to write this piece to throw some light on the ways to modify headers in automated testing.

Meet Image Optimization, Addy Osmani’s brand new practical guide to optimizing and delivering high-quality images on the web. From formats and compression to delivery and maintenance: everything in one single 528-pages book.

Selenium is widely used as a test automation framework for end-to-end testing of web applications. It was developed in 2004. Initially, Selenium IDE was being used but it only supported firefox then Selenium RC was developed to enable cross-browser testing. Now the Selenium WebDriver is being used as it provides support for the mobile experience and dynamic websites. It mimics a real user interacting with the webpage.

Advantages Of Using Selenium WebDriver

Limitations Of Selenium WebDriver

As discussed, the ability to modify headers helps tremendously in testing applications but the Selenium WebDriver doesn’t support it, and they don’t plan to include it lately.

This article focuses on various approaches to modify headers in a Selenium automation setup.

Approach 1: Selenium Wire (Python)

Selenium Wire extends Selenium Python bindings to give you access to the underlying requests made by the browser. You author your code in the same way as you do with Selenium, but you get extra APIs for inspecting requests and responses and making changes to them on the fly.

It allows modifying requests and responses on the go using interceptors. It can also block requests, mock responses, add request parameters in the URL.

Adding Request Headers

Duplicate header names are permitted in an HTTP request, so before setting the replacement header you must first delete the existing header using del otherwise two headers of the same name will exist.

Adding Response Headers

A response interceptor should accept two arguments, one for the originating request and one for the response.

Limitations

Approach 2: Loading Browser Extension In Selenium Which Can Modify Headers

There are tools like Requestly which is a one-stop tool to debug & modify network requests. Requestly allows users to Modify Headers, Redirect URLs, Switch hosts, Mock API response, Delay network requests, Insert custom scripts, etc. It provides ready to go npm package which is a wrapper around the extension enabling the use of extension in Selenium. It supports Chrome, Firefox and Edge.

Using Requestly In Selenium

Requestly allows you to do a lot more things other than just modifying headers. All these things can be set up within a web interface and the shared list can be loaded into the application which allows the user’s flexibility to edit the rules easily in a web application.

To Install: npm i @requestly/selenium

A Modify Headers Rule can be created at app.requestly.io/rules after installing the extension.

After creating the URL, click on the Share button to generate a link for the URL.

The URL for the above created sharedList is here.

This URL is a Requestly rule which adds Access-Control-Allow-Origin header to all requests.

This rule can be used in the Selenium WebDriver using the sharedList URL which is described below:

require("chromedriver");
const { Builder } = require("selenium-webdriver");
const chrome = require("selenium-webdriver/chrome");
const { getRequestlyExtension, importRequestlySharedList } = require("@requestly/selenium");

const options = new chrome.Options().addExtensions(getRequestlyExtension("chrome"));
const driver = new Builder()
    .forBrowser("chrome")
    .setChromeOptions(options)
    .build();

// Imports Rules in Selenium using Requestly sharedList feature
// importRequestlySharedList(driver, <sharedList_URL>);

importRequestlySharedList(driver, 'https://app.requestly.io/rules/#sharedList/1626984924247-Adding-Headers-Example');

More information can be found here.

Limitations

Approach 3: Using Puppeteer

Puppeteer is a Node library developed by Google which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

When talking about browser automation setup, Selenium automatically comes to the mind but since the advent of Puppeteer it is widely being used for web scraping. It offers more control over chrome than Selenium probably due to google’s support to it. Also due to the same reason, it rules out the need for an external driver to run the browser.

Usage

Modifying Headers

The page.setExtraHTTPHeaders(headers) method can be used to set headers.

It can also modify and remove existing headers. Once requests are intercepted using a combination of page.setRequestInterception(true) and page.on().

Modifying network headers happens to be a very powerful tool in a testing environment. There’s a lot more to it than what can be covered in this article. I have tried to cover some of the simplest methods to modify headers in UI Automation Testing.

Every method has some advantages based on the use case and language you use. Here are some of the preferences:

Hope this article gives you insight on modifying headers in an automated web application testing setup.