cthulhu/test/README

CTHULHU TEST HARNESS:
------------------

This document provides a very brief description of the Cthulhu test
harness.  The Cthulhu test harness is not meant to be our final answer to
testing, but it helps fill the gap until more general testing for
GNOME is avaialble.


MAIN IDEAS:
-----------

The main ideas behind the Cthulhu test harness are as follows:

* We try to make sure there is a textual description of a test that
  someone can follow by hand when sitting behind the machine.  The 
  description contains the goal of the test, the procedure to follow,
  and the expected results.  We try to keep the description of the
  tests here:

  http://cvs.gnome.org/viewcvs/*checkout*/cthulhu/docs/doc-set/cthulhu.html#TESTINGPLAN

* Each test is intended to be small and test for a specific feature
  of Cthulhu and/or for a specific bug that has been been found and
  fixed.

* Cthulhu provides the ability to send a textual description of what it
  is sending to speech and braille.  This allows for the recording
  of output, thus allowing for comparison to results in the future.

* The harness merely consists of playing back pre-recorded keystrokes
  and comparing Cthulhu's output to a previously recorded successful run
  (where 'success' means Cthulhu output the correct information).


DIRECTORY LAYOUT:
-----------------

The Cthulhu regression tests contained in this directory are laid out as
follows:

./harness:    test harness scripts

./keystrokes: keystroke and otyher files for playing back tests

./results:    reference results containing accurate output from Cthulhu
              when the test harness runall script was run using the
              keystrokes


PREREQUISITES:
--------------

The main prerequisite for running the tests is that you've checked the
cthulhu module out from the GNOME git repository.  If you want to do
regression testing to compare the test results with a known 'correct'
set of results, you need to create an 'cthulhu' user on your machine and
run the tests while logged in as that user.  It's best if you check
the cthulhu module out directly in the home directory of the user running
the tests.


MAIN FILES:
-----------

./harness/runall.sh: The ./harness directory contains two main
  scripts: runall.sh and runone.sh.  The runall.sh script is the main
  script to use and will execute all the tests and will place the
  results in a directory whose name is of the form YYYY-MM-DD_HH:MM:SS
  (e.g., 2006-11-29_20:21:41).

./harness/runone.sh: The runone.sh script allows a test creator to
  run just one test.  The parameters are:  <*.keys file> <app-name>
  where the *.keys file contains keystrokes and <app-name> is the
  name of the application to run with the given *.keys file.

./harness/user-settings.py.in: contains the default Cthulhu configuration
  settings.  The primary thing this file does is disable the use of
  real speech and braille and instead sends a text form of the speech
  and braille to a log file.  Note that there are also facilities to
  specify a custom user-settings file for each keystroke file so as to
  allow the testing of Cthulhu settings (e.g., echo by word).

./keystrokes/*: The ./keystrokes directory consists of a set of
  directories, where each directory name has significant meaning
  and is used to determine which app the test harness uses for
  testing.  The directory name is typically the name of a binary 
  on the path.  For example, there is a ./keystrokes/gtk-demo
  directory, and the files under this directory are for testing Cthulhu
  using the gtk-demo application.  The other type of directory name
  is one that matches the output of 'uname', and is used to contain
  platform specific tests (e.g., Ctrl+Esc for Solaris versus Alt+F1
  for Linux).

  Under each ./keystrokes/* directory, there will be one or more
  *.keys files.  These contain keystrokes recorded using the 
  record_keystrokes.py utility (see RECORDING TESTS below).  In
  addition, there may be an option *.settings file for each *.keys
  file.  The *.settings file is an Cthulhu settings file to use
  specifically for the *.keys file, and is used to help test
  specific Cthulhu features such as word echo, key echo, etc.

./results/*: The ./results directory should contain a set of
  directories that matches those in the ./keystrokes directory.
  Under each of the ./results/* directores are *.cthulhu files
  containing the reference speech and braille output from a
  known successful run of Cthulhu using the associated *.keys file.


RECORDING KEYSTROKES:
---------------------

To record a test, you should first start the application to be tested
and leave it in its initial startup state.  For example, if you want
to write a test for gedit, just start 'gedit' and don't do anything in
it yet.

In a separate terminal window, run cthulhu's 'record_keystrokes.py'
module and immediately give focus back to the window of the
application you are testing.  You can run the record_keystrokes.py
module as follows (the 'sleep 5' gives you time to put focus back on
the app):

  sleep 5; python ~/cthulhu/src/tools/record_keystrokes.py > myfile.keys

Then, interact with the application and press F12 when you are done.
The F12 tells record_keystrokes.py to quit.  If you want to do a rough
verification that the recorded file is OK, you can run the
'sanity_check.py' module:

  python ~/cthulhu/src/tools/sanity_check.py < myfile.keys

NOTE: You need to be somewhat careful about recording tests.  There
are cases where some keystrokes do not make it to the AT-SPI, such as
when applications do keyboard grabs.  In these cases, you need to find
a different way to accomplish what you're trying to do.  In addition,
be aware that some applications also embed time-based and
username-specific stuff in them.  They also sometimes modify their
menus based upon past use (e.g., a list of recently used documents in
the 'File' menu).  In these cases, you should try to avoid navigating
through these areas so as to avoid inconsistent output from run to
run.

NOTE ALSO: the test harness automatically starts and kills the
application to be tested.  As such, you usually do not need to record
keystrokes to exit the application being tested, unless you happen to
be writing a test for that, of course.  Some apps, such as Nautilus
and Open Office, tend to spawn off or connect to background processes.
In order to ensure these types of applications quit, you actually do
need to make sure the keystrokes to quit these applications are
contained in every keystroke file.


RUNNING THE HARNESS:
--------------------

To run the harness, merely run the runall.sh script.  It will run
through all the keystrokes and compare the results to the known
results.


KNOWN ISSUES:
-------------

* Solaris and Linux use different keycodes.  The keystroke files
  currently are recorded on Ubuntu.  The work needed here might be to
  create a directory called ./keystrokes_solaris parallel to the
  ./keystrokes directory, and then specify this directory via the "-k"
  command line option to the runall.sh script.

* The OS-specific (e.g., 'uname') portions of the harness really do not
  work yet.  Instead, the keystroke files are always played regardless
  of platform.  We need to work this out.

* OpenOffice behaves oddly when popping up dialogs: there seems to be
  a variable delay on these dialogs and the played keystrokes do not
  always seem to get to the right window.  You'll be able to tell this
  because OpenOffice will not quit when the test is completed.  For
  now, the workaround is to keep rerunning the tests via the runone.sh
  script until the tests complete.