rose stem

Rose User Guide: rose stem

rose stem

Introduction

This part of the Rose user guide explains how to use the rose stem testing system, and the motivation behind it.

rose stem requires the use of FCM as it requires some of the version control information.

Motivation

Why do we test code?

Motivation II

Most people would answer something along the lines of "so we know it works".

However, this is really asking two related but separate questions.

  • Does the code do what I meant it to do?
  • Does the code do anything I didn't mean it to do?

Motivation III

Answering the first question may involve writing a bespoke test and checking the results. The second question can at least partially be answered by using an automated testing system which runs predefined tasks and presents the answers. rose stem is a system for doing this.

N.B. When writing tests for new code, they should be added to the testing system so that future developers can be confident that they haven't broken the new functionality.

Rose Stem

There are two components in rose stem:

  • rose stem - the command line tool which executes an appropriate suite.
  • rose ana - a Rose task app which can compare the result of a task against a control.

We will describe each in turn. It is intended that a test suite lives alongside the code in the same version-controlled project, which should encourage developers to update the test suite when they update the code. This means that the test suite will always be a valid test of the code it is accompanying.

Running a suite with rose stem

The rose stem command is essentially a wrapper to rose suite-run, which accepts some additional arguments and converts them to Jinja2 variables which the suite can interpret.

These arguments are:

  • --source - specifies a source tree to include in a suite.
  • --group - specifies a group of tasks to run.

A group is a set of Rose tasks which together test a certain configuration of a program.

The --source argument

The source argument provides a set of Jinja2 variables which can then be included in any compilation tasks in a suite. You can specify multiple --source arguments on the command line. For example:

rose stem --source=/path/to/workingcopy --source=fcm:other_project_tr@head

Each source tree is associated with a project (via an fcm command) when rose stem is run on the command line. This project name is then used in the construction of the Jinja2 variable names.

Source II

Each project has a Jinja2 variable SOURCE_FOO where FOO is the project name. This contains a space-separated list of all sourcetrees belonging to that project, which can then be given to an appropriate build task in the suite so it builds those source trees.

Similarly, a HOST_SOURCE_FOO variable is also provided. This is identical to SOURCE_FOO except any working copies have the local hostname prepended. This is to assist building on remote machines.

The first source specified must be a working copy which contains the rose stem suite. The suite is expected to be in a subdirectory named rose-stem off the top of the working copy. This source is used to generate three additional variables:

  • SOURCE_FOO_BASE - the base directory of the project
  • HOST_SOURCE_FOO_BASE - the base directory of the project with the hostname prepended if it is a working copy
  • SOURCE_FOO_REV - the revision of the project (if any)

These settings override the variables in the rose-suite.conf file.

Source III

These should allow the use of configuration files which control the build process inside the working copy, e.g. you can refer to:

      {{HOST_SOURCE_FOO_BASE}}/fcm-make/configs/machine.cfg{{SOURCE_FOO_REV}}

If you omit the source argument, rose stem defaults to assuming

rose stem --source=.

viz. that the current directory is part of the working copy which should be added as a source tree.

Source IV

The project to which a source tree belongs is normally automatically determined using FCM commands. However, in the case where the source tree is not a valid FCM URL, or where you wish to assign it to another project, you can specify this using the --source argument:

rose stem --source=foo=/path/to/source

assigns the URL /path/to/source to the foo project, so the variables SOURCE_FOO and SOURCE_FOO_BASE will be set to /path/to/source.

The --group argument

The group argument is used to provide a Pythonic list of groups in the variable RUN_NAMES which can then be looped over in a suite to switch sets of tasks on and off.

Each --group argument adds another group to the list. For example:

rose stem --group=mygroup --group=myothergroup

runs two groups named mygroup and myothergroup with the current working copy. The suite will then interpret these into a set of tasks which build with the given source tree(s), run the program, and compare the output.

The --task argument

The --task argument is provided as a synonym for --group. Depending on how exactly the rose stem suite works users may find one of these arguments more intuitive to use than the other.

Comparing output with rose ana

Any task beginning with rose_ana_ will be interpreted by Rose as a rose ana task, and run through the rose ana task app.

A rose ana rose-app.conf file contains a series of blocks; each one describing a different analysis task to perform. A common task which rose ana is used for is to compare output contained in different files (e.g. from a new test versus previous output from a control). The analysis modules which provide these tasks are flexible and able to be provided by the user; however there is one built-in module inside rose ana itself.

rose ana II

An example based on the built-in grepper module:

  [ana:grepper.FilePattern(Compare data from myfile)]
  pattern='data value:(\d+)'
  files=/data/kgo/myfile
       =../run.1/myfile

This tells rose ana to scan the contents of the file ../run.1/myfile (which is relative to the rose ana task's work directory) and the contents of /data/kgo/myfile for the specified regular expression. Since the pattern contains a group (in parentheses) so it is the contents of this group which will be compared between the two files. The grepper.FilePattern analysis task can optionally be given a "tolerance" option for matching numeric values, but without it the matching is expected to be exact. If the pattern or group contents do not match the task will return a failure.

rose ana III

It is possible to add additional analysis modules to rose ana by placing an appropriately formatted python file in one of the following places (in order of precedence):

  • The ana sub-directory of the rose ana app
  • The ana sub-directory of the suite
  • Any other directory which is accessible to the process running rose ana and is specified in the method-path variable in the rose ana section of the rose.conf file.

The only module provided with Rose can be found at lib/python/rose/apps/ana_builtin/grepper.py and it provides the following analysis tasks and options:

  • SingleCommandStatus: Runs a shell command, passing or failing depending on the exist status of that command.
    • files - a newline-separated list of filenames (optional) which may appear in the command.
    • command - the command to run; if it contains Python style format specifiers these will be expanded using the list of files above (if provided).
    • kgo_file - if the list of files above was provided gives the (0-based) index of the file holding the "kgo" or "control" output for use with the comparisons database (if active).
  • SingleCommandPattern: Runs a shell command, then passes or fails depending on the presence of a given regular expression in the command's stdout.
    • files - a newline-separated list of filenames (optional) which may appear in the command.
    • command - the command to run; if it contains Python style format specifiers these will be expanded using the list of files above (if provided).
    • kgo_file - if the list of files above was provided gives the (0-based) index of the file holding the "kgo" or "control" output for use with the comparisons database (if active).
    • pattern - the regular expression to search for in the stdout from the command.
  • FilePattern: Compares the contents of files using a regular expression.
    • files - a newline-separated list of filenames to be compared.
    • kgo_file - if the list of files above was provided gives the (0-based) index of the file holding the "kgo" or "control" output for use with the comparisons database (if active).
    • pattern - the regular expression to search for in the files. The expression should include one or more capture groups; each of these will be compared between the files any time the pattern occurs.
    • tolerance - by default the above comparisons will be compared exactly, but if this (optional) argument is specified they will be converted to float values and compared according to the given tolerance. If this tolerance ends in % it will be interpreted as a relative tolerance (otherwise absolute).
  • FileCommandPattern: Uses a regular expression to compares the contents of the stdout resulting from a given command applied to two or more files.
    • files - a newline-separated list of filenames to be substitued into the command.
    • kgo_file - if the list of files above was provided gives the (0-based) index of the file holding the "kgo" or "control" output for use with the comparisons database (if active).
    • pattern - the regular expression to search for in the stdout. The expression should include one or more capture groups; each of these will be compared between the stdout any time the pattern occurs.
    • tolerance - by default the above comparisons will be compared exactly, but if this (optional) argument is specified they will be converted to float values and compared according to the given tolerance. If this tolerance ends in % it will be interpreted as a relative tolerance (otherwise absolute).
    • command - the command to run; it should contain a Python style format specifier to be expanded using the list of files above.

rose ana IV

The format for analysis modules themselves is relatively simple; the easiest route to understanding how they should be arranged is likely to look at the built-in grepper module. But the key concepts are as follows. To be recognised as a valid analysis module, the Python file must contain at least one class which inherits and extends rose.apps.rose_ana.AnalysisTask e.g:

from rose.apps.rose_ana import AnalysisTask

class CustomAnalysisTask(AnalysisTask):
    """My new custom analysis task."""
    def run_analysis(self):
        print self.options
        print self.parent

Assuming the above was saved in a file called custom.py and placed into a folder suitable for analysis modules this would allow a rose ana app to specify:

[ana:custom.CustomAnalysisTask(Example rose-ana test)]
option1 = 5
option2 = test of rose ana
option3 = .true.

All options specified above will be processed by rose ana into a dictionary and attached to the running analysis class instance as the options attribute, and a reference to the parent RoseAnaApp instance will be attached as the parent attribute. The result of the above examples would therefore print:

{'option2': 'test of rose ana', 'option3': '.true.', 'option1': '5'}
<class 'rose.apps.rose_ana.RoseAnaApp'>

Therefore from this point the AnalysisTask class could access the options via their names. Note however that all options are treated as strings - it is up to the class to apply any required checks or conversions.

The rose ana comparison database

In addition to performing the comparisons each of the rose ana tasks in the suite can be configured to append some key details about any comparisons performed to an sqlite database kept in the suite's log directory (at log/rose-ana-comparisons.db).

This is intended to provide a quick means to interrogate the suite for information about the status of any comparisons it has performed. There are 2 tables present in the suite which contain the following:

rose ana db II

  • tasks (TABLE) Contains an entry for each rose ana task, using the following columns:
    • task_name (TEXT) The exact name of the rose ana task.
    • completed (INT) Set to 1 when the task starts performing its comparisons then updated to 0 when the task has completed (Note: task success is not related to the success/failed state of the comparisons).
    The intention of this table is to detect if any rose ana tasks have failed unexpectedly (or are still running).

rose ana db III

  • comparisons Contains an entry for each individual comparison from every rose ana task, using the following columns:
    • app_task (TEXT) A composite of the exact name of the rose ana task and the comparison section name from the app defintion (in brackets).
    • kgo_file (TEXT) The full path to the file specified as the KGO file in the app definition.
    • suite_file (TEXT) The full path to the file specified as the active test output in the app definition.
    • status (TEXT) The status of the task (one of " OK ", "FAIL" or "WARN").
    • comparison (TEXT) Additional details which may be provided about the comparison
    The intention of this table is to provide a record of which files were compared by which tasks, how they were compared and what the result of the comparison was.

rose ana db IV

The database is entirely optional; by default is will not be produced; if it is required it can be activated by setting the kgo-database option in the rose-ana section of the Rose configuration to .true..

Note that the system does not provide any direct methods for working with or interrogating the database - since there could be various reasons for doing so, and there may be other suite-design factors to consider. Users are therefore expected to provide this functionality separately based on their specific needs.

Summary

From within a working copy, running rose stem is simple. Just run

rose stem --group=groupname
replacing the groupname with the desired task. rose stem should then automatically pick up the working copy and run the requested tests on it.

For further information about writing your own rose stem test suite, please see the advanced tutorial on the subject.