CTHULHU TEST HARNESS:
------------------
This document provides a very brief description of the Cthulhu test
harness. The Cthulhu test harness is not meant to be our final answer to
testing, but it helps fill the gap until more general testing for
GNOME is avaialble.
MAIN IDEAS:
-----------
The main ideas behind the Cthulhu test harness are as follows:
* We try to make sure there is a textual description of a test that
someone can follow by hand when sitting behind the machine. The
description contains the goal of the test, the procedure to follow,
and the expected results. We try to keep the description of the
tests here:
http://cvs.gnome.org/viewcvs/*checkout*/cthulhu/docs/doc-set/cthulhu.html#TESTINGPLAN
* Each test is intended to be small and test for a specific feature
of Cthulhu and/or for a specific bug that has been been found and
fixed.
* Cthulhu provides the ability to send a textual description of what it
is sending to speech and braille. This allows for the recording
of output, thus allowing for comparison to results in the future.
* The harness merely consists of playing back pre-recorded keystrokes
and comparing Cthulhu's output to a previously recorded successful run
(where 'success' means Cthulhu output the correct information).
DIRECTORY LAYOUT:
-----------------
The Cthulhu regression tests contained in this directory are laid out as
follows:
./harness: test harness scripts
./keystrokes: keystroke and otyher files for playing back tests
./results: reference results containing accurate output from Cthulhu
when the test harness runall script was run using the
keystrokes
PREREQUISITES:
--------------
The main prerequisite for running the tests is that you've checked the
cthulhu module out from the GNOME git repository. If you want to do
regression testing to compare the test results with a known 'correct'
set of results, you need to create an 'cthulhu' user on your machine and
run the tests while logged in as that user. It's best if you check
the cthulhu module out directly in the home directory of the user running
the tests.
MAIN FILES:
-----------
./harness/runall.sh: The ./harness directory contains two main
scripts: runall.sh and runone.sh. The runall.sh script is the main
script to use and will execute all the tests and will place the
results in a directory whose name is of the form YYYY-MM-DD_HH:MM:SS
(e.g., 2006-11-29_20:21:41).
./harness/runone.sh: The runone.sh script allows a test creator to
run just one test. The parameters are: <*.keys file> <app-name>
where the *.keys file contains keystrokes and <app-name> is the
name of the application to run with the given *.keys file.
./harness/user-settings.py.in: contains the default Cthulhu configuration
settings. The primary thing this file does is disable the use of
real speech and braille and instead sends a text form of the speech
and braille to a log file. Note that there are also facilities to
specify a custom user-settings file for each keystroke file so as to
allow the testing of Cthulhu settings (e.g., echo by word).
./keystrokes/*: The ./keystrokes directory consists of a set of
directories, where each directory name has significant meaning
and is used to determine which app the test harness uses for
testing. The directory name is typically the name of a binary
on the path. For example, there is a ./keystrokes/gtk-demo
directory, and the files under this directory are for testing Cthulhu
using the gtk-demo application. The other type of directory name
is one that matches the output of 'uname', and is used to contain
platform specific tests (e.g., Ctrl+Esc for Solaris versus Alt+F1
for Linux).
Under each ./keystrokes/* directory, there will be one or more
*.keys files. These contain keystrokes recorded using the
record_keystrokes.py utility (see RECORDING TESTS below). In
addition, there may be an option *.settings file for each *.keys
file. The *.settings file is an Cthulhu settings file to use
specifically for the *.keys file, and is used to help test
specific Cthulhu features such as word echo, key echo, etc.
./results/*: The ./results directory should contain a set of
directories that matches those in the ./keystrokes directory.
Under each of the ./results/* directores are *.cthulhu files
containing the reference speech and braille output from a
known successful run of Cthulhu using the associated *.keys file.
RECORDING KEYSTROKES:
---------------------
To record a test, you should first start the application to be tested
and leave it in its initial startup state. For example, if you want
to write a test for gedit, just start 'gedit' and don't do anything in
it yet.
In a separate terminal window, run cthulhu's 'record_keystrokes.py'
module and immediately give focus back to the window of the
application you are testing. You can run the record_keystrokes.py
module as follows (the 'sleep 5' gives you time to put focus back on
the app):
sleep 5; python ~/cthulhu/src/tools/record_keystrokes.py > myfile.keys
Then, interact with the application and press F12 when you are done.
The F12 tells record_keystrokes.py to quit. If you want to do a rough
verification that the recorded file is OK, you can run the
'sanity_check.py' module:
python ~/cthulhu/src/tools/sanity_check.py < myfile.keys
NOTE: You need to be somewhat careful about recording tests. There
are cases where some keystrokes do not make it to the AT-SPI, such as
when applications do keyboard grabs. In these cases, you need to find
a different way to accomplish what you're trying to do. In addition,
be aware that some applications also embed time-based and
username-specific stuff in them. They also sometimes modify their
menus based upon past use (e.g., a list of recently used documents in
the 'File' menu). In these cases, you should try to avoid navigating
through these areas so as to avoid inconsistent output from run to
run.
NOTE ALSO: the test harness automatically starts and kills the
application to be tested. As such, you usually do not need to record
keystrokes to exit the application being tested, unless you happen to
be writing a test for that, of course. Some apps, such as Nautilus
and Open Office, tend to spawn off or connect to background processes.
In order to ensure these types of applications quit, you actually do
need to make sure the keystrokes to quit these applications are
contained in every keystroke file.
RUNNING THE HARNESS:
--------------------
To run the harness, merely run the runall.sh script. It will run
through all the keystrokes and compare the results to the known
results.
KNOWN ISSUES:
-------------
* Solaris and Linux use different keycodes. The keystroke files
currently are recorded on Ubuntu. The work needed here might be to
create a directory called ./keystrokes_solaris parallel to the
./keystrokes directory, and then specify this directory via the "-k"
command line option to the runall.sh script.
* The OS-specific (e.g., 'uname') portions of the harness really do not
work yet. Instead, the keystroke files are always played regardless
of platform. We need to work this out.
* OpenOffice behaves oddly when popping up dialogs: there seems to be
a variable delay on these dialogs and the played keystrokes do not
always seem to get to the right window. You'll be able to tell this
because OpenOffice will not quit when the test is completed. For
now, the workaround is to keep rerunning the tests via the runone.sh
script until the tests complete.
