Using Tox with Travis CI to Test Django Apps

Being a fan of good testing, I'm always trying to find ways to improve testing on various projects. Travis CI and Coveralls are really nice ways to set up continuous integration for your open-source projects. A couple months ago I finally started hearing grumblings about tox and how everyone was it using for their Python test automation. Every time I'd try to wrap my head around it, something always eluded me, so this week I finally decided to dive in head first and see if I could get to the bottom of it and how it could improve my current integration setup.

Headless Django Environments

Most of my open-source Python projects have been developed from the beginning to be installable modules for use with tools like pip. With Django apps, this tends to be a little more difficult as Django development environments generally assumes you have a project that contains a lot boilerplate (manage.py, project/settings.py, project/urls.py), but some of this stuff is just clutter in an installable package. After trying several patterns, I found the one used by James Socol on several projects, using fabric to wrap my development enviroment and easily provide some of the common commands we use manage.py for, but without all the clutter of a full Django project. I've also created a repository that's intended to be used as a Django project template, to help with setting out a project like this. It even includes some of the setup I'm about to discuss. You can get that template over at Github.

Preparing tox

Start by installing tox with a simple pip install tox. Then create the tox.ini file in the base of your project. The first section you care about is configuring tox itself.

[tox]
envlist = django15, django16

The envlist directive tells tox which environments to run by default when calling it from the CLI. Tox comes with a few built-in environments (py26, py27, py31, py33, etc.) for testing different Python versions, but in my case I care about testing different Django versions, so I'll be building my own environments.

[testenv]
commands = django-admin.py test
setenv =
    DJANGO_SETTINGS_MODULE=test_app.settings
    PYTHONPATH={toxinidir}

testenv is the set of default environment settings that all named environments will inherit from. Here I'm telling it that the default command I'm going to use is django-admin.py test to run tests. And setup environment variables that are usually handled by manage.py. Tox also allows for substituting variables, in this case {toxinidir} is simply the path of the directory which contains the tox.ini file. It also allows for doing substitutions based on lookups within the ini file. By defining a custom section, we can define settings that can be reused and extended within our environments.

[base]
deps =
    mox
    nose
    django-nose

Here I'm defining some dependencies that I'm going to add to in each environment. In my case, I prefer Nose and Mox to their standard library counterparts.

[testenv:django15]
deps =
    django>=1.5, <1.6
    {[base]deps}

[testenv:django16]
deps =
    django>=1.6, <1.7
    {[base]deps}

This creates two new test environments, named django15 and django16, and adds the dependency to install the proper version of Django, 1.5 and 1.6 respectively. It also uses the section lookup to append the deps from the base section we defined above. Now, when you return to the CLI and run tox, you should see it setup virtualenvs for each environment and run your tests under them. That's all it take to get it running. Anyone can now clone your repository and simply run tox to confirm that tests are passing. This is particularly useful if they intend to contribute back to the project.

Adding Coverage

Now, as I mentioned before, I want to also use Coveralls to show coverage stats as well. To do that I just need to define another environment.

[testenv:coverage]
commands =
    coverage run --branch --omit={envdir}/*,test_app/*.py,*/migrations/*.py {envbindir}/django-admin.py test
    coveralls
deps =
    coverage
    coveralls
    {[testenv:django16]deps}

This time I'm running coverage run with branch coverage, ignoring migrations and my test app and running the django-admin.py in the enviroments bin directory, followed by the coveralls reporting script. We also have a couple of additional dependencies to add to the django16 deps.

Running Tox on Travis

With tox configured to handle the details, Travis just needs to know about the different environments. Here's what the .travis.yaml looks like.

language: python
install:
  - pip install tox 
script:
  - tox
env:
  - TOXENV=django15
  - TOXENV=django16
  - TOXENV=coverage

This tells Travis to install tox and to run the tox command. Adding the TOXENV environment variable makes it easy to specify which testenv to run.Travis runs it's tasks based on the build matrix of environment variables and language versions, allowing us to run each testenv asynchronously and reporting the results for each version separately.