181 lines
7.5 KiB
Plaintext
181 lines
7.5 KiB
Plaintext
CTHULHU TEST HARNESS:
|
|
------------------
|
|
|
|
This document provides a very brief description of the Cthulhu test
|
|
harness. The Cthulhu test harness is not meant to be our final answer to
|
|
testing, but it helps fill the gap until more general testing for
|
|
GNOME is avaialble.
|
|
|
|
|
|
MAIN IDEAS:
|
|
-----------
|
|
|
|
The main ideas behind the Cthulhu test harness are as follows:
|
|
|
|
* We try to make sure there is a textual description of a test that
|
|
someone can follow by hand when sitting behind the machine. The
|
|
description contains the goal of the test, the procedure to follow,
|
|
and the expected results. We try to keep the description of the
|
|
tests here:
|
|
|
|
http://cvs.gnome.org/viewcvs/*checkout*/cthulhu/docs/doc-set/cthulhu.html#TESTINGPLAN
|
|
|
|
* Each test is intended to be small and test for a specific feature
|
|
of Cthulhu and/or for a specific bug that has been been found and
|
|
fixed.
|
|
|
|
* Cthulhu provides the ability to send a textual description of what it
|
|
is sending to speech and braille. This allows for the recording
|
|
of output, thus allowing for comparison to results in the future.
|
|
|
|
* The harness merely consists of playing back pre-recorded keystrokes
|
|
and comparing Cthulhu's output to a previously recorded successful run
|
|
(where 'success' means Cthulhu output the correct information).
|
|
|
|
|
|
DIRECTORY LAYOUT:
|
|
-----------------
|
|
|
|
The Cthulhu regression tests contained in this directory are laid out as
|
|
follows:
|
|
|
|
./harness: test harness scripts
|
|
|
|
./keystrokes: keystroke and otyher files for playing back tests
|
|
|
|
./results: reference results containing accurate output from Cthulhu
|
|
when the test harness runall script was run using the
|
|
keystrokes
|
|
|
|
|
|
PREREQUISITES:
|
|
--------------
|
|
|
|
The main prerequisite for running the tests is that you've checked the
|
|
cthulhu module out from the GNOME git repository. If you want to do
|
|
regression testing to compare the test results with a known 'correct'
|
|
set of results, you need to create an 'cthulhu' user on your machine and
|
|
run the tests while logged in as that user. It's best if you check
|
|
the cthulhu module out directly in the home directory of the user running
|
|
the tests.
|
|
|
|
|
|
MAIN FILES:
|
|
-----------
|
|
|
|
./harness/runall.sh: The ./harness directory contains two main
|
|
scripts: runall.sh and runone.sh. The runall.sh script is the main
|
|
script to use and will execute all the tests and will place the
|
|
results in a directory whose name is of the form YYYY-MM-DD_HH:MM:SS
|
|
(e.g., 2006-11-29_20:21:41).
|
|
|
|
./harness/runone.sh: The runone.sh script allows a test creator to
|
|
run just one test. The parameters are: <*.keys file> <app-name>
|
|
where the *.keys file contains keystrokes and <app-name> is the
|
|
name of the application to run with the given *.keys file.
|
|
|
|
./harness/user-settings.py.in: contains the default Cthulhu configuration
|
|
settings. The primary thing this file does is disable the use of
|
|
real speech and braille and instead sends a text form of the speech
|
|
and braille to a log file. Note that there are also facilities to
|
|
specify a custom user-settings file for each keystroke file so as to
|
|
allow the testing of Cthulhu settings (e.g., echo by word).
|
|
|
|
./keystrokes/*: The ./keystrokes directory consists of a set of
|
|
directories, where each directory name has significant meaning
|
|
and is used to determine which app the test harness uses for
|
|
testing. The directory name is typically the name of a binary
|
|
on the path. For example, there is a ./keystrokes/gtk-demo
|
|
directory, and the files under this directory are for testing Cthulhu
|
|
using the gtk-demo application. The other type of directory name
|
|
is one that matches the output of 'uname', and is used to contain
|
|
platform specific tests (e.g., Ctrl+Esc for Solaris versus Alt+F1
|
|
for Linux).
|
|
|
|
Under each ./keystrokes/* directory, there will be one or more
|
|
*.keys files. These contain keystrokes recorded using the
|
|
record_keystrokes.py utility (see RECORDING TESTS below). In
|
|
addition, there may be an option *.settings file for each *.keys
|
|
file. The *.settings file is an Cthulhu settings file to use
|
|
specifically for the *.keys file, and is used to help test
|
|
specific Cthulhu features such as word echo, key echo, etc.
|
|
|
|
./results/*: The ./results directory should contain a set of
|
|
directories that matches those in the ./keystrokes directory.
|
|
Under each of the ./results/* directores are *.cthulhu files
|
|
containing the reference speech and braille output from a
|
|
known successful run of Cthulhu using the associated *.keys file.
|
|
|
|
|
|
RECORDING KEYSTROKES:
|
|
---------------------
|
|
|
|
To record a test, you should first start the application to be tested
|
|
and leave it in its initial startup state. For example, if you want
|
|
to write a test for gedit, just start 'gedit' and don't do anything in
|
|
it yet.
|
|
|
|
In a separate terminal window, run cthulhu's 'record_keystrokes.py'
|
|
module and immediately give focus back to the window of the
|
|
application you are testing. You can run the record_keystrokes.py
|
|
module as follows (the 'sleep 5' gives you time to put focus back on
|
|
the app):
|
|
|
|
sleep 5; python ~/cthulhu/src/tools/record_keystrokes.py > myfile.keys
|
|
|
|
Then, interact with the application and press F12 when you are done.
|
|
The F12 tells record_keystrokes.py to quit. If you want to do a rough
|
|
verification that the recorded file is OK, you can run the
|
|
'sanity_check.py' module:
|
|
|
|
python ~/cthulhu/src/tools/sanity_check.py < myfile.keys
|
|
|
|
NOTE: You need to be somewhat careful about recording tests. There
|
|
are cases where some keystrokes do not make it to the AT-SPI, such as
|
|
when applications do keyboard grabs. In these cases, you need to find
|
|
a different way to accomplish what you're trying to do. In addition,
|
|
be aware that some applications also embed time-based and
|
|
username-specific stuff in them. They also sometimes modify their
|
|
menus based upon past use (e.g., a list of recently used documents in
|
|
the 'File' menu). In these cases, you should try to avoid navigating
|
|
through these areas so as to avoid inconsistent output from run to
|
|
run.
|
|
|
|
NOTE ALSO: the test harness automatically starts and kills the
|
|
application to be tested. As such, you usually do not need to record
|
|
keystrokes to exit the application being tested, unless you happen to
|
|
be writing a test for that, of course. Some apps, such as Nautilus
|
|
and Open Office, tend to spawn off or connect to background processes.
|
|
In order to ensure these types of applications quit, you actually do
|
|
need to make sure the keystrokes to quit these applications are
|
|
contained in every keystroke file.
|
|
|
|
|
|
RUNNING THE HARNESS:
|
|
--------------------
|
|
|
|
To run the harness, merely run the runall.sh script. It will run
|
|
through all the keystrokes and compare the results to the known
|
|
results.
|
|
|
|
|
|
KNOWN ISSUES:
|
|
-------------
|
|
|
|
* Solaris and Linux use different keycodes. The keystroke files
|
|
currently are recorded on Ubuntu. The work needed here might be to
|
|
create a directory called ./keystrokes_solaris parallel to the
|
|
./keystrokes directory, and then specify this directory via the "-k"
|
|
command line option to the runall.sh script.
|
|
|
|
* The OS-specific (e.g., 'uname') portions of the harness really do not
|
|
work yet. Instead, the keystroke files are always played regardless
|
|
of platform. We need to work this out.
|
|
|
|
* OpenOffice behaves oddly when popping up dialogs: there seems to be
|
|
a variable delay on these dialogs and the played keystrokes do not
|
|
always seem to get to the right window. You'll be able to tell this
|
|
because OpenOffice will not quit when the test is completed. For
|
|
now, the workaround is to keep rerunning the tests via the runone.sh
|
|
script until the tests complete.
|