Writing Regression Tests
From K-3D
Overview
For information on running existing tests, see Running Regression Tests.
This article is a whirlwhind overview on writing tests for K-3D's CMake/CTest based build system. Because we are relatively new to test-based coding, our test system (and this documentation) are in flux as we work-out best practices.
Tests
In a nutshell, a test is simply a process that when run returns a zero exit code to indicate test success, or a non-zero exit code to indicate test failure.
Tests can be binary executables written in C++ or scripts written in any of the scripting languages K-3D supports - either K3DScript or Python as of this writing. The current test suite includes examples of all three. In general, we use Python for tests wherever possible since it offers the greatest functionality and ease-of-use while providing the necessary platform portability. K3DScript is primarily seen in user interface tutorials that are based on "recorded" user interaction, providing useful "smoke testing" of the user interface. C++ tests are often unit tests that verify the implementation of a class or function.
Recording User Interface Tests
At the highest level, it is possible to record user interaction with K-3D's NGUI graphical user interface, and play it back as a test. Recording a test in this fashion is the preferred mechanism for user-reported bugs, since it creates an objective record of how to reproduce an issue without relying on the accuracy or thoroughness of the reporter. The recording mechanism writes UI data to a file, flushing it after every event - so the output file is always valid, even if the program crashes during recording. Tests recorded in this way are recorded as Python scripts, so they can combine programmable logic with user actions if needed - as an example, a test could be modified after recording to react to the runtime environment, tests can be modified in a text editor, and multiple scripts can be combined.
To create a test based on UI recording, do the following:
- Start K-3D
- Immediately select the Scripting > Record Test Case menu item.
- You will be prompted to choose a filename for the recorded test. Typically, tests are recorded to the top-level tests/ directory in the source tree.
- Once a filename has been selected, a small window opens to notify you that recording is in operation.
- Leave the recording window open, and interact with K-3D as you normally would, going through the operations necessary to duplicate the bug / test the intended functionality.
- When you are finished recording, close the recording window.
- Exit K-3D.
To play-back a test case while K-3D is running, choose the Scripting > Play menu item and select the recorded test script.
To make a recorded test ngui.foo.py part of the regression test suite, add the following to tests/CMakeLists.txt:
K3D_ADD_TEST(ngui.foo.py PYTHON NGUI)
... the K3D_ADD_TEST macro adds a test that will run the given script. Note that by convention we start UI-related tests with "ngui." so that they can be easily selected when running a subset of available tests.
Creating Pipeline Tests
Many tests are designed to exercise plugin functionality and should not require user interaction at all. In these cases, tests are written by hand using Python. Typically, each of these tests creates a new document, creates a small pipeline within the document, and retrieves data from the pipeline to be tested against known-good references. There are many examples of this type of tests in the tests/ directory. A small Python module, tests/testing.py provides utility functions that reduce the boilerplate code when writing such tests. As an example:
#python
import testing
setup = testing.setup_mesh_source_test("PolyCube")
testing.assert_valid_mesh(setup.output_mesh)
testing.assert_solid_mesh(setup.output_mesh)
testing.mesh_comparison(setup.document, setup.source.get_property("output_mesh"), "mesh.source.PolyCube", 1)
In this example, a PolyCube mesh source is created. Three tests are performed on the output of the mesh source - one for internal data validity, one to ensure that the mesh is a solid polyhedron, and a final comparison for equality with a known-good reference mesh. The reference mesh will be the file tests/meshes/mesh.source.PolyCube.reference.k3d in this case. For all tests a Python exception is raised to signal test failure.
Other, similar scripts are used to test mesh modifiers and bitmap-related plugins. Our ultimate goal is to achieve 100% test coverage of all plugins.
Adding Tests to the Build
Once a test (either C++ or scripted) has been written, it must be added to the build system. This is done with the cmake ADD_TEST() command. See k3dsdk/tests/CMakeLists.txt for examples of how this is done with C++ tests. For scripted tests, the command-line to start K-3D and run a test is relatively complex, so several cmake macros have been defined in tests/CMakeLists.txt to simplify the task.
Once a test has been added to the build and committed to subversion, it will be automatically run each night by the K-3D Dashboard.
Notable Filesystem Locations
- k3dsdk/tests/ - contains several compiled C++ tests that exercise the K-3D SDK libraries
- modules/test/ - K-3D plugin module that provides several plugins that have been written for use in testing:
- tests/ - the main collection of regression tests, most written in Python
- tests/testing.py - a Python module providing helpful functions for writing scripted tests
- tests/bitmaps - contains reference images for use during testing
- tests/meshes - contains reference meshes stored in K-3D (XML) format, for use during testing
