Make sure all files that should be present are present.
This commit is contained in:
Storm Dragon
180
test/README
Normal file
180
test/README
Normal file
@ -0,0 +1,180 @@
|
||||
ORCA TEST HARNESS:
|
||||
------------------
|
||||
|
||||
This document provides a very brief description of the Cthulhu test
|
||||
harness. The Cthulhu test harness is not meant to be our final answer to
|
||||
testing, but it helps fill the gap until more general testing for
|
||||
GNOME is avaialble.
|
||||
|
||||
|
||||
MAIN IDEAS:
|
||||
-----------
|
||||
|
||||
The main ideas behind the Cthulhu test harness are as follows:
|
||||
|
||||
* We try to make sure there is a textual description of a test that
|
||||
someone can follow by hand when sitting behind the machine. The
|
||||
description contains the goal of the test, the procedure to follow,
|
||||
and the expected results. We try to keep the description of the
|
||||
tests here:
|
||||
|
||||
http://cvs.gnome.org/viewcvs/*checkout*/cthulhu/docs/doc-set/cthulhu.html#TESTINGPLAN
|
||||
|
||||
* Each test is intended to be small and test for a specific feature
|
||||
of Cthulhu and/or for a specific bug that has been been found and
|
||||
fixed.
|
||||
|
||||
* Cthulhu provides the ability to send a textual description of what it
|
||||
is sending to speech and braille. This allows for the recording
|
||||
of output, thus allowing for comparison to results in the future.
|
||||
|
||||
* The harness merely consists of playing back pre-recorded keystrokes
|
||||
and comparing Cthulhu's output to a previously recorded successful run
|
||||
(where 'success' means Cthulhu output the correct information).
|
||||
|
||||
|
||||
DIRECTORY LAYOUT:
|
||||
-----------------
|
||||
|
||||
The Cthulhu regression tests contained in this directory are laid out as
|
||||
follows:
|
||||
|
||||
./harness: test harness scripts
|
||||
|
||||
./keystrokes: keystroke and otyher files for playing back tests
|
||||
|
||||
./results: reference results containing accurate output from Cthulhu
|
||||
when the test harness runall script was run using the
|
||||
keystrokes
|
||||
|
||||
|
||||
PREREQUISITES:
|
||||
--------------
|
||||
|
||||
The main prerequisite for running the tests is that you've checked the
|
||||
cthulhu module out from the GNOME git repository. If you want to do
|
||||
regression testing to compare the test results with a known 'correct'
|
||||
set of results, you need to create an 'cthulhu' user on your machine and
|
||||
run the tests while logged in as that user. It's best if you check
|
||||
the cthulhu module out directly in the home directory of the user running
|
||||
the tests.
|
||||
|
||||
|
||||
MAIN FILES:
|
||||
-----------
|
||||
|
||||
./harness/runall.sh: The ./harness directory contains two main
|
||||
scripts: runall.sh and runone.sh. The runall.sh script is the main
|
||||
script to use and will execute all the tests and will place the
|
||||
results in a directory whose name is of the form YYYY-MM-DD_HH:MM:SS
|
||||
(e.g., 2006-11-29_20:21:41).
|
||||
|
||||
./harness/runone.sh: The runone.sh script allows a test creator to
|
||||
run just one test. The parameters are: <*.keys file> <app-name>
|
||||
where the *.keys file contains keystrokes and <app-name> is the
|
||||
name of the application to run with the given *.keys file.
|
||||
|
||||
./harness/user-settings.py.in: contains the default Cthulhu configuration
|
||||
settings. The primary thing this file does is disable the use of
|
||||
real speech and braille and instead sends a text form of the speech
|
||||
and braille to a log file. Note that there are also facilities to
|
||||
specify a custom user-settings file for each keystroke file so as to
|
||||
allow the testing of Cthulhu settings (e.g., echo by word).
|
||||
|
||||
./keystrokes/*: The ./keystrokes directory consists of a set of
|
||||
directories, where each directory name has significant meaning
|
||||
and is used to determine which app the test harness uses for
|
||||
testing. The directory name is typically the name of a binary
|
||||
on the path. For example, there is a ./keystrokes/gtk-demo
|
||||
directory, and the files under this directory are for testing Cthulhu
|
||||
using the gtk-demo application. The other type of directory name
|
||||
is one that matches the output of 'uname', and is used to contain
|
||||
platform specific tests (e.g., Ctrl+Esc for Solaris versus Alt+F1
|
||||
for Linux).
|
||||
|
||||
Under each ./keystrokes/* directory, there will be one or more
|
||||
*.keys files. These contain keystrokes recorded using the
|
||||
record_keystrokes.py utility (see RECORDING TESTS below). In
|
||||
addition, there may be an option *.settings file for each *.keys
|
||||
file. The *.settings file is an Cthulhu settings file to use
|
||||
specifically for the *.keys file, and is used to help test
|
||||
specific Cthulhu features such as word echo, key echo, etc.
|
||||
|
||||
./results/*: The ./results directory should contain a set of
|
||||
directories that matches those in the ./keystrokes directory.
|
||||
Under each of the ./results/* directores are *.cthulhu files
|
||||
containing the reference speech and braille output from a
|
||||
known successful run of Cthulhu using the associated *.keys file.
|
||||
|
||||
|
||||
RECORDING KEYSTROKES:
|
||||
---------------------
|
||||
|
||||
To record a test, you should first start the application to be tested
|
||||
and leave it in its initial startup state. For example, if you want
|
||||
to write a test for gedit, just start 'gedit' and don't do anything in
|
||||
it yet.
|
||||
|
||||
In a separate terminal window, run cthulhu's 'record_keystrokes.py'
|
||||
module and immediately give focus back to the window of the
|
||||
application you are testing. You can run the record_keystrokes.py
|
||||
module as follows (the 'sleep 5' gives you time to put focus back on
|
||||
the app):
|
||||
|
||||
sleep 5; python ~/cthulhu/src/tools/record_keystrokes.py > myfile.keys
|
||||
|
||||
Then, interact with the application and press F12 when you are done.
|
||||
The F12 tells record_keystrokes.py to quit. If you want to do a rough
|
||||
verification that the recorded file is OK, you can run the
|
||||
'sanity_check.py' module:
|
||||
|
||||
python ~/cthulhu/src/tools/sanity_check.py < myfile.keys
|
||||
|
||||
NOTE: You need to be somewhat careful about recording tests. There
|
||||
are cases where some keystrokes do not make it to the AT-SPI, such as
|
||||
when applications do keyboard grabs. In these cases, you need to find
|
||||
a different way to accomplish what you're trying to do. In addition,
|
||||
be aware that some applications also embed time-based and
|
||||
username-specific stuff in them. They also sometimes modify their
|
||||
menus based upon past use (e.g., a list of recently used documents in
|
||||
the 'File' menu). In these cases, you should try to avoid navigating
|
||||
through these areas so as to avoid inconsistent output from run to
|
||||
run.
|
||||
|
||||
NOTE ALSO: the test harness automatically starts and kills the
|
||||
application to be tested. As such, you usually do not need to record
|
||||
keystrokes to exit the application being tested, unless you happen to
|
||||
be writing a test for that, of course. Some apps, such as Nautilus
|
||||
and Open Office, tend to spawn off or connect to background processes.
|
||||
In order to ensure these types of applications quit, you actually do
|
||||
need to make sure the keystrokes to quit these applications are
|
||||
contained in every keystroke file.
|
||||
|
||||
|
||||
RUNNING THE HARNESS:
|
||||
--------------------
|
||||
|
||||
To run the harness, merely run the runall.sh script. It will run
|
||||
through all the keystrokes and compare the results to the known
|
||||
results.
|
||||
|
||||
|
||||
KNOWN ISSUES:
|
||||
-------------
|
||||
|
||||
* Solaris and Linux use different keycodes. The keystroke files
|
||||
currently are recorded on Ubuntu. The work needed here might be to
|
||||
create a directory called ./keystrokes_solaris parallel to the
|
||||
./keystrokes directory, and then specify this directory via the "-k"
|
||||
command line option to the runall.sh script.
|
||||
|
||||
* The OS-specific (e.g., 'uname') portions of the harness really do not
|
||||
work yet. Instead, the keystroke files are always played regardless
|
||||
of platform. We need to work this out.
|
||||
|
||||
* OpenOffice behaves oddly when popping up dialogs: there seems to be
|
||||
a variable delay on these dialogs and the played keystrokes do not
|
||||
always seem to get to the right window. You'll be able to tell this
|
||||
because OpenOffice will not quit when the test is completed. For
|
||||
now, the workaround is to keep rerunning the tests via the runone.sh
|
||||
script until the tests complete.
|
||||
Reference in New Issue
Block a user