Testing legacy systems - Smoke Testing

In any company that is more than a few years old, there is very likely some system we could call “legacy”. I am not going to try to define what a legacy system is, but we can all agree that once we get to work on one, we tend to find ourselves asking “Why?” very often. The problem is that usually the answer is unknown but we have to work with it anyway. Following lack of any documentation answering these questions is a lack of any tests that would point us in the right direction and at the very least increase our confidence that we did not break anything.

At Football Radar we have one system in particular that feels very 'legacy', which we call "The Webapp". Very often, after making a change, we test it by running it locally and/or in a staging environment and just click around the important pages, making sure that they still work. This is a kind of ‘smoke test’. It’s not the best testing method but slightly better than just throwing it over the wall and praying that it works!

A repetitive task just asks to be automated - my plan was simple:

  1. Automate test environment creation
  2. Make a list of endpoints to test
  3. Write a test to hit all these endpoints and check their status codes
  4. Run this on each commit, together with the other tests
1. Automate test environment creation

This was the most time consuming task. The solution sounds simple - make a Docker image that will run the app. Execution was a bit more complicated in practice because of the nature of the app. It has a lot of dependencies (the base image is ~200MB and the final image is ~2GB). We are not going to post our Dockerfile here as this is very specific to us and you would not benefit much from it.

The difficulty here was that we didn’t have documented anywhere what dependencies are really required - I had simply rewritten our Ansible scripts as Dockerfiles. At this stage the most important thing is just to make it build and run successfully. Once you have it working (that is, at least serving the home page) you can continue. Later, when you will have the whole smoke test setup in place, you can revisit your Dockerfile and adjust it if you missed something and some tests are failing.

Additionally, as we need to have a redis instance running next to our webapp instance, I wrote a simple Docker compose configuration

version: '2'  
services:  
  webapp:
    build: .
    ports:
      - "15000:80"
    depends_on:
      - redis
      - doctrine_cache
  redis:
    image: redis
  doctrine_cache:
    image: redis

And we are ready to go - we can start our webapp with just one command docker-compose up --build

2. Make a list of endpoints to test

This was the easy part- I just compiled a list of all the pages we had been testing manually so far, plus any more that we could think of, and I made a collection containing them all. Some URLs needed to be tested with different parameters, so we just do some simple string replacement as you can see below. These are all GET endpoints - testing POST and other verbs will come later - this is just the MVP.

NOTE: router and context come from Symfony\Component\Routing and are here just to help with generating fully qualified URLs.

    function getUrls() {
       $paths = [
            ['path' => '/mvc/widget', 'params' => []],
            ['path' => '/mvc/players/index?team_id=%s', 'params' => [13956]],
            ['path' => '/mvc/players/index?team_id=%s', 'params' => [444]],
            ['path' => '/mvc/players/index?team_id=%s', 'params' => [2090]],
            /* … */
        ];
        foreach ($paths as $path) {
            $this->context->setBaseUrl(sprintf($path['path'], ...$path['params']));
            $urls[] = $this->router->generate('home', [], RouterInterface::ABSOLUTE_URL);
        }
        return $urls;
   }
3. Write a test to hit all these endpoints and check their status codes

Our app is written in PHP and we use PHPUnit to run these tests, but obviously it could be anything. The task is simple - just take the collection defined above, iterate over it and make the request with an http client. Once you have the response, check that it has a success code. Simple. Below you can find our code as an example:

<?php

namespace FootballRadar\AppBundle\Tests;

use GuzzleHttp\Client;  
use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;  
use Symfony\Component\HttpFoundation\Response;  
use Symfony\Component\Routing\RequestContext;  
use Symfony\Component\Routing\RouterInterface;

/**
 * @group SmokeTest
 */
class SmokeTest extends KernelTestCase {

    /** @var RouterInterface */
    private $router;

    /** @var Client */
    private $client;

    private $headers = [];

    /** @var RequestContext */
    private $context;

    public function setUp() {
        self::bootKernel();
        $container = self::$kernel->getContainer();
        $this->router = $container->get('router');
        $this->context = new RequestContext('/web', 'GET', 'localhost', 'http', 15000);
        $this->router->setContext($this->context);
        $this->client = new Client(['cookies' => true]);
    }

    public function testUrls() {
        $this->logIn();
        foreach ($this->getUrls() as $url) {
            $response = $this->client->request('GET', $url, ['allow_redirects' => false]);
            $this->assertEquals(
                Response::HTTP_OK,
                $response->getStatusCode(),
                sprintf('Failed at %s response code: %d', $url, $response->getStatusCode())
            );
        }
    }

    }

    private function getUrls() {
         /** see above **/
    }


    private function logIn() {
        // creates php session, we need it to exists before we attempt to login
        $this->client->request('GET', $this->router->generate('home', [], RouterInterface::ABSOLUTE_URL));
        $response = $this->client->request(
            'POST',
            $this->router->generate('login_check', [], RouterInterface::ABSOLUTE_URL),
            [
                'form_params' => [
                    '_username' => 'test_user',
                    '_password' => 'test_user',
                    'client' => 0,
                ],
            ]
        );
        $this->headers = $response->getHeaders();
    }
}

Not very complicated but I guess a few words of explanation are required.

We could use PHPUnit’s dataprovider to provide URLs, and have whole URLs hardcoded instead of generating them. This would allow us to run each URL as separate test case which might be desirable in some ways, but I decided against it for two reasons:
We have a few more routes that I didn’t show here that are defined using Symfony routing, and we use a router generator to get them, just like we get the login URL in login() method.
We didn’t feel the need to run each URL as a separate test case, and running them together allows us to login just once instead of needing to perform authentication for each URL. In the future, as the test evolves, this requirement may change - we may want to test one URL for multiple users with different permissions and expect a 403 in some cases, for example - but that is future concern.

4. Run this on each commit with the other tests

We were actually already running our tests on commit, so we just needed to modify this configuration to include starting Docker, running the smoke tests and stopping Docker. We use Jenkins pipeline for this. Here is our Jenkinsfile:

#!groovy

node {  
    def workspace = pwd()

    try {
        notifyBuild('STARTED')

        stage("Checkout") {
            checkout scm
        }

        stage("Prepare directories") {
            sh "rm -rf ${workspace}/app/logs/* ${workspace}/app/cache/* ${workspace}/vendor"
        }

        stage("Create environment file") {
            sh "cp ${workspace}/docker/environment.php ${workspace}/environment.php"
        }

        stage("Install composer and vendors") {
            sh "${workspace}/composer-install.sh"
            sh "${workspace}/composer.phar install -n --no-suggest"
            sh "${workspace}/app/console cache:warmup --env=test"
        }

        stage("Build and start docker") {
            env.COMPOSE_HTTP_TIMEOUT=120
            sh "docker-compose -f ${workspace}/docker-compose.yml up --build -d"
        }

        stage("Tests") {
            parallel(
                    "Syntax check": {
                        node {
                            sh "find ${workspace} -path ${workspace}/vendor -prune -o -name '*.php' -exec php -l {} \\;"
                        }
                    },
                    "Unit tests small": {
                        node {
                            sh "${workspace}/vendor/bin/phpunit -c ${workspace}/app/phpunit.xml.dist --group small"
                        }
                    },
                    "Unit tests medium ": {
                        node {
                            sh "${workspace}/vendor/bin/phpunit -c ${workspace}/app/phpunit.xml.dist --group medium"
                        }
                    },
                    "Unit tests large": {
                        node {
                            sh "${workspace}/vendor/bin/phpunit -c ${workspace}/app/phpunit.xml.dist --group large"
                        }
                    },
                    "Unit tests other": {
                        node {
                            sh "${workspace}/vendor/bin/phpunit -c ${workspace}/app/phpunit.xml.dist --exclude-group small,medium,large,SmokeTest"
                        }
                    },
                    "Smoke tests": {
                        node {
                            // we wait for docker
                            sh "sleep 60"
                            sh "${workspace}/vendor/bin/phpunit -c ${workspace}/app/phpunit.xml.dist --group SmokeTest"
                        }
                    }
            )
        }

        notifyBuild('SUCCESSFUL')
    } catch (e) {
        notifyBuild('FAILED')
        throw e
    } finally {
        stage("Stop docker") {
            sh "docker-compose -f ${workspace}/docker-compose.yml down"
        }
    }
}

def notifyBuild(String buildStatus = 'STARTED') {  
    // build status of null means successful
    buildStatus =  buildStatus ?: 'SUCCESSFUL'

    // Default values
    def colorCode = '#d9534f'
    def message = "<${env.JOB_URL}|${env.JOB_NAME}> - <${env.BUILD_URL}|${env.BUILD_DISPLAY_NAME}>\n" +
                    "Commits:\n ${getCommits()}"
    // Override default values based on build status
    if (buildStatus == 'STARTED') {
        colorCode = '#f0ad4e'
    } else if (buildStatus == 'SUCCESSFUL') {
        colorCode = '#1ecc00'
    }

    // Send notifications
    slackSend (color: colorCode, message: message, channel: '#webapp', token: "${SLACK_WEBAPP}")
}

@NonCPS
def getCommits() {  
    def changeLogSets = currentBuild.changeSets
    def commits = ''
    for (int i = 0; i < changeLogSets.size(); i++) {
        def entries = changeLogSets[i].items
        for (int j = 0; j < entries.length; j++) {
            def c = entries[j]
            commits += "<https://github.com/footballradar/Web/commit/${c.commitId}|${c.msg}> by ${c.author}\n"
        }
    }
    return commits
}

Our docker needs a bit of time to get ready and start serving requests, so that’s why we have sleep there for 60 seconds. The section at the bottom will post a message to our Slack when build starts and when it finishes, notifying us of success or failure.

With all these elements together, we have a bit more confidence now when making changes to our legacy application. It is far from perfect, but it is much better than nothing. Primary limitation is it does not test any business logic, page may load successfully but content may be wrong. I guess for our PHP application this will allow us to catch at least some of the errors that would be found at compile time in some other languages.

Next steps we plan to do are:
  • Set up a dedicated handcrafted database to run in a Docker instance alongside the application - currently it is connecting to a dev instance that is a daily copy of our production database. This is the most important next step but also the most complicated, as we need to find an easy way of maintaining it and keeping it up to date.
  • Once we have the above-mentioned database, we can write some functional tests, potentially with Cucumber. We haven’t decided this yet but it’s an idea we are considering.
  • Set up dynamic port assignment, so it is possible to run more than one job at the time. Then maybe we could run it on Mesos - our container orchestration platform.

When we have solved these remaining problems, we will write a follow-up post about them.