Split Test Traffic Distribution with Nginx

Great programming is worthless if you’re building the wrong thing. Testing early and often can help validate your assumptions to keep a project or company on the right track. This type of testing can often require traffic distribution strategies for siphoning users into test groups.

Nginx configured as a load balancer can accomodate complex distribution logic and provides a simple solution for most split test traffic distribution needs.

Full App Tests

In straightforward cases, when we’re testing an entirely new version of an application that runs on a distinct server, we can create a load balancer to proxy requests for the domain, passing a desired portion of requests to the test server.

The Nginx configuration for this load balancer could be as simple as:

http {
  upstream appServer {
    ip_hash;
    server old.app.com weight=9;
    server new.app.com;
  }
  server {
    listen 80;
    location / {
      proxy_pass http://appServer;
    }
  }
}

In this example, Nginx is configured to choose between passing requests to one of two app servers. (old.app.com and new.app.com could be listed as IP addresses instead if you’re into that sort of thing.)

Session Persistence

ip_hash is used for session persistence (to avoid having visitors see two different versions of the app on subsequent requests.)

Distribution Weighting

Most test cases require that just a small fraction of requests be passed to the test version of the app. Here a weight parameter is used to adjust how frequently requests are passed to the new version of the app that is under test. A weight of 9 has the same effect as having 9 seperate entries for this old.app.com server. When the routing decision is being made, Nginx will choose one server from the (effective) 10 server entries. In this case, the new app server will be passed 10% of requests.

Partial App Tests

Split testing a partial replacement for an existing app is more complicated than a full replacement. If we simply switched requests between servers with a partial replacement, requests could be made to the new app server for portions of the existing app that do not exist in the version under test. 404 time.

Subdomain Partial Redirection Strategy

In these complex cases, it may be best to make your test version available via a subdomain, preserving the naked domain for accessing necessary portions of the old application.

We recently tested a replacement for a homepage and several key pages for an existing application. Our Nginx configuration looked like that below. (Or here if you would prefer an uncommented version.)

# Define a cluster to which you can proxy requests. In this case, we will be
# proxying requests to just a single server: the original app server.
# See http://nginx.org/en/docs/http/ngx_http_upstream_module.html
upstream app.com {
  server old.app.com;
}
# Assign to a variable named $upstream_variant a psuedo-randomly chosen value,
# with "test" being assigned 10% of the time, and "original" assigned the
# remaining 90% of the time. (This is group determination for requests not
# already containing a group cookie.)
# See http://nginx.org/en/docs/http/ngx_http_split_clients_module.html
split_clients "app${remote_addr}${http_user_agent}${date_gmt}" $upstream_variant {
 10% "test";
  * "original";
}
# Assign to a variable named $updstream_group a variable mapped from the the
# value present in the group cookie. If the cookie's value is present, preserve
# the existing value. If it is not, assign to the value of the $upstream_variant.
# See http://nginx.org/en/docs/http/ngx_http_map_module.html
# Note: the value of cookies is available via the $cookie_ variables
# (i.e. $cookie_my_cookie_name will return the value of the cookie named 'my_cookie_name').
map $cookie_split_test_version $upstream_group {
  default $upstream_variant;
  "test" "test";
  "original" "original";
}
# Assign to a variable named $internal_request a value indicating whether or
# not the given request originates from an internal IP address. If the request
# originates from an IP within the range defined by 4.3.2.1 to 4.3.2.254, assign 1.
# Otherwise assign 0.
# See http://nginx.org/en/docs/http/ngx_http_geo_module.html
geo $internal_request {
  ranges;
  4.3.2.1-4.3.2.254 1;
  default 0;
}
server {
  listen 80 default_server;
  listen [::]:80 default_server ipv6only=on;
  server_name app.com www.app.com;
  # For requests made to the root path (in this case, the hompage):
 
  location = / {
    # Set a cookie containing the selected group as it's value. Expire after
    # 6 days (the length of our test).
 
    add_header Set-Cookie "split_test_version=$upstream_group;Path=/;Max-Age=518400;";
    # Requests by default are not considered candidates for the test group.
 
    set $test_group 0;
    # If the request has been randomly selected for the test group, it is now
    # a candidate for redirection to the test site.
 
    if ($upstream_group = "test") {
      set $test_group 1;
    }
    # Regardless of the group determination, if the request originates from
    # an internal IP it is not a candidate for the test group.
 
    if ($internal_request = 1) {
      set $test_group 0;
    }
    # Redirect test group candidate requests to the test subdomain.
    # See http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return
 
    if ($test_group = 1) {
      return 302 http://new.app.com/;
      break;
    }
    # Pass all remaining requests through to the old application.
    # See http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass
 
    proxy_pass http://app.com;
  }
  # For requests to all other paths:
 
  location / {
    # Pass the request through to the old application.
    proxy_pass http://app.com;
  }
}

Switch Location

Partial split tests require a specific location from which to assign the test group and begin the test. In this case, the root location (homepage) is used. Requests to the root path are assigned a test group and redirected or passed through as appropriate. Requests to all other locations are passed through to the original app server.

Session Persistence (and Expiration)

In this case, we use a cookie (add_header Set-Cookie ...) to establish session persistence. When a request is assigned to either the test or original group, this group is recorded in a cookie which is accessed upon subsequent requests to ensure a consistent experience for the duration of the test.

Since our split test was set to last less than 6 days, we set a six day expiration on the cookie.

IP Filtering

In many cases, split tests require the filtering of certain parties (in this case the company owner of the application) to avoid skewing test results. In this case, we use Nginx’s geo to determine whether or not the request originates from the company’s internal network. Later on this determination is used to direct the request straight through to the original version of the app.

302 Redirection

Since most of the original application must remain accessible, instead of passing test group traffic through to the new server, we instead 302 redirect the request to the new application subdomain. This allows the user to seemlessly switch between viewing content provided by both the new and original applications.

Nginx is a terrific tool for distributing traffic for split tests. It’s stable, it’s blazingly fast, and configurations for typical use cases are prevalent online. More complex configuration can be accomplished after just a couple hours exploring the documentation. Give Nginx a shot next time you split test!