Running Regression Tests

From K-3D

Overview

For information on how to create regression tests, see Writing Regression Tests.

The K-3D Regression Test Suite includes a growing, sophisticated collection of compiled and scripted tests that can be run to test software quality, catch errors introduced during development, detect platform incompatibilities, and calculate code coverage metrics. Test results can be interpreted by the individual developer, or publicly shared via the K-3D Dashboard - an automated, interactive website for collecting and displaying test results via a standard web browser. In good open-source fashion, any K-3D community member can run the test suite and submit results to the Dashboard, providing developers with results from a more diverse collection of platforms, and providing rapid feedback when code changes have introduced problems. In the future, the Dashboard's code coverage feature will make it easy to browse through the sources, seeing code coverage on a line-by-line basis with a web browser.

Running Tests

Note: All of the following commands are run from your build directory, once ccmake has been run with K3D_ENABLE_TESTING to ON:

To list all available regression tests:

$ ctest -N

To run all regression tests:

$ ctest

To run regression tests that match a regular expression, such as "all bitmap source (reader) tests":

$ ctest -R bitmap.source

To skip regression tests that match a regular expression, such as "all user interface tests":

$ ctest -E ngui

To run a specific test with verbose output (including output generated by the test):

$ ctest -V -R bitmap.source.JPEGBitmapReader

Submitting Tests to the Dashboard

You can run all of the regression tests on your current sources and submit the results to the K-3D Dashboard as an experimental dashboard. This is useful when there are tests failing and you want to show other developers your test results, without having to commit your modifications to subversion:

$ ctest -D Experimental

... note that you can combine the -D option with other options, if you only wish to submit a single test, a group of tests, etc.

You can also run a continuous dashboard, which will update your current sources from subversion, and if there were changes, configure, build, run the regression tests, and submit the results to the K-3D Dashboard. If subversion hasn't changed, the continuous dashboard returns immediately without doing anything. Thus, you could safely run the following from a periodic cron-job, and it would only submit results if the sources have actually changed:

$ ctest -D Continuous

A nightly dashboard will update your current sources to 2:00 AM US Mountain Time, configure, build, run the regression test suite, and submit the results to the K-3D dashboard. These actions are performed whether subversion has changed or not. In this way the K-3D dashboard always has a fresh set of builds every morning:

$ ctest -D Nightly

Running Tests Offscreen

Many of the regression tests use K-3D's tutorial system, which takes control of the mouse pointer and interacts with the graphical user interface directly. It is a major inconvenience if you're trying to do other work while the tests run, as you fight K-3D for control of your mouse pointer! Fortunately, users running X Windows can run the regression tests offscreen, using an instance of Xvfb, the virtual X server. A sample script for running the regression tests using Xvfb is as follows:

#!/bin/sh

killall Xvfb
Xvfb :89 -ac -screen 0 1024x768x24+32 &

export DISPLAY=":89"
export LD_PRELOAD=/usr/lib/nvidia/libGL.so.1.2.xlibmesa

ctest -V -S "~/src/k3d-nightly/cmake/dashboards/standard-debug-nightly.cmake,CTEST_BINARY_DIRECTORY;~/k3d-nightly"

killall Xvfb

Highlights of this script:

• Assumes that you have downloaded a source tree to ~/src/k3d-nightly.
• Assumes that you have created a build directory ~/build/k3d-nightly.
• Runs Xvfb with a 1024 x 768, 24-bit-depth screen, on display 89 (an arbitrary choice).
• Uses LD_PRELOAD to ensure that the K-3D binaries are run using the Xorg / Mesa OpenGL implementation instead of any proprietary OpenGL (such as nvidia). Unfortunately, hardware-accelerated OpenGL implementations seem to cause problems with Xvfb, including crashing the server. Even more unfortunately, finding the correct implementation libraries will require trial-and-error, since their location and naming will vary from platform-to-platform and even across distribution versions.
• Runs the entire set of regression tests using the standard-debug-nightly ctest script provided in the K-3D sources, which does a complete reconfigure and rebuild from scratch every night. Note that you might need to copy-and-modify this script if there are specific dependency or configuration issues on your system.

To see the contents of the virtual framebuffer while tests are running:

$ xwd -root -display :89 | xwud