Conditional blank referrer with Nginx

In this article, we’ll be showing a common use case of the referrers in the Nginx configuration and cover a use case from the real-life project our teams worked on. The task at hand is to allow a blank referrer in a scenario when a specific user-agent (Android device) makes an HTTP request – a conditional blank referrer with Nginx.

Pre-requirement: Nginx installed.

What is the HTTP referrer used for

When we want to make sure that traffic is not being drained by crawlers, scrapers, etc. (although, this is a much harder task than presented here and requires a more detailed approach to protecting online content) we’d usually want to define allowed domains from which the request should be passed through. Since this might have been a mouthful, let’s simply like this:

• when we open a website our browser fills in the page with images and other content
• images and other static content come from somewhere (it can be the same server the website is on or some other location) by the browser “calling in” this content in order to render the page
• when the browser makes these requests for necessary content it passes information about the domain from which the requests were made (in this case the domain of the website in question)
• finally, a server that is supposed to be serving these files (images for example) returned the content and we got our page fully rendered

Now, let’s imagine that (and this is a real-life scenario) we want to make sure that only visitors of our website have the right to use the bandwidth needed to transfer all the images, etc. To do this, using Nginx, we can define something called “valid_referrers” and list the allowed domains (or just a single domain):

valid_referers bluegrid.io;
		if ($invalid_referer) {
			return 403;
		}

Edge case

Now, the edge case we are covering is when there’s an expected scenario where a specific user-agent can access the content even without a referrer at all – a blank referrer. The blank referrer scenario happens when a device accesses content directly, without a proxy, website, or other means of being passed from another domain.

What we need to do is check if the traffic comes from the expected user-agent and bypass the valid_referrers directive:

location / {
		set $check "1";
		if ($http_user_agent ~ (Android)) {                                  
			set $check "0";
		}
		if ($http_referer ~ ^$) {                                  
			set $invalid_referer $check;
		}
		valid_referers bluegrid.io;
		if ($invalid_referer) {
			return 403;
		}

Test solution

Test access with valid referrer:

~$ curl -I storage.bluegrid.io/ --referer http://bluegrid.io

HTTP/1.1 200 OK

Test access with the blank referrer and expected user-agent:

~$ curl -I storage.bluegrid.io/ -A "Android"

HTTP/1.1 200 OK

CURL without referrer and without user agent:

~$ curl -I storage.bluegrid.io/

HTTP/1.1 403 Forbidden