Test and Demo Functions
***********************

Octave includes a number of functions to allow the integration of testing and demonstration code in the source code of the functions themselves.

Test Functions
==============

 -- test: NAME
 -- test: NAME quiet|normal|verbose
 -- test: ("NAME", "quiet|normal|verbose", FID)
 -- test: ("NAME", "quiet|normal|verbose", FNAME)
 -- SUCCESS =: test (...)
 -- [N, NMAX, NXFAIL, NBUG, NSKIP, NRTSKIP, NREGRESSION] =: test (...)
 -- [CODE, IDX] =: test ("NAME", "grabdemo")
 -- test: ([], "explain", FID)
 -- test: ([], "explain", FNAME)

     Perform built-in self-tests from the first file in the loadpath matching NAME.

     test can be called in either command or functional form.  The exact operation of test is determined by a combination of mode (interactive or batch), reporting level ("quiet", "normal", "verbose"), and whether a logfile or summary output variable is used.

     The default mode when test is called from the command line is interactive.  In this mode, tests will be run until the first error is encountered, or all tests complete successfully.  In batch mode, all tests are run regardless of any failures, and the results are collected for reporting.  Tests which require user interaction, i.e., demo blocks, are never run in batch mode.

     Batch mode is enabled by either 1) specifying a logfile using the third argument FNAME or FID, or 2) requesting an output argument such as SUCCESS, N, etc.

     The optional second argument determines the amount of output to generate and which types of tests to run.  The default value is "normal". Requesting an output argument will suppress printing the final summary message and any intermediate warnings, unless verbose reporting is enabled.

     "quiet"
          Print a summary message when all tests pass, or print an error with the results of the first bad test when a failure occurs.  Don't run tests which require user interaction.

     "normal"
          Display warning messages about skipped tests or failing xtests during test execution. Print a summary message when all tests pass, or print an error with the results of the first bad test when a failure occurs.  Don't run tests which require user interaction.

     "verbose"
          Display tests before execution.  Print all warning messages.  In interactive mode, run all tests including those which require user interaction.

     The optional third input argument specifies a logfile where results of the tests should be written.  The logfile may be a character string (FNAME) or an open file descriptor ID (FID).  To enable batch processing, but still print the results to the screen, use stdout for FID.

     When called with just a single output argument SUCCESS, test returns true if all of the tests were successful.  If called with more than one output argument then the number of successful tests (N), the total number of tests in the file (NMAX), the number of xtest failures (NXFAIL), the number of tests failed due known bugs (NBUG), the number of tests skipped due to missing features (NSKIP), the number of tests skipped due to run-time conditions (NRTSKIP), and the number of regressions (NREGRESSION) are returned.

     Example

          test sind
          =>
          PASSES 5 out of 5 tests

          [n, nmax] = test ("sind")
          =>
          n =  5
          nmax =  5

     Additional Calling Syntaxes

     If the second argument is the string "grabdemo", the contents of any built-in demo blocks are extracted but not executed.  The text for all code blocks is concatenated and returned as CODE with IDX being a vector of positions of the ends of each demo block.  For an easier way to extract demo blocks from files, see example.

     If the second argument is "explain" then NAME is ignored and an explanation of the line markers used in test output reports is written to the file specified by FNAME or FID.

     assert, fail, demo, example, error

test scans the named script file looking for lines which start with the identifier %!.  The prefix is stripped off and the rest of the line is processed through the Octave interpreter.  If the code generates an error, then the test is said to fail.

Since eval() will stop at the first error it encounters, you must divide your tests up into blocks, with anything in a separate block evaluated separately.  Blocks are introduced by valid keywords like test, function, or assert immediately following %!. A block is defined by indentation as in Python.  Lines beginning with %!<whitespace> are part of the preceding block.

For example:

     %!test error ("this test fails!")
     %!test "test doesn't fail.  it doesn't generate an error"

When a test fails, you will see something like:

       ***** test error ("this test fails!")
     !!!!! test failed
     this test fails!

Generally, to test if something works, you want to assert that it produces a correct value.  A real test might look something like

     %!test
     %! A = [1, 2, 3; 4, 5, 6]; B = [1; 2];
     %! expect = [ A ; 2*A ];
     %! get = kron (B, A);
     %! if (any (size (expect) != size (get)))
     %!   error ("wrong size: expected %d,%d but got %d,%d",
     %!          size (expect), size (get));
     %! elseif (any (any (expect != get)))
     %!   error ("didn't get what was expected.");
     %! endif

To make the process easier, use the assert function.  For example, with assert the previous test is reduced to:

     %!test
     %! A = [1, 2, 3; 4, 5, 6]; B = [1; 2];
     %! assert (kron (B, A), [ A; 2*A ]);

assert can accept a tolerance so that you can compare results absolutely or relatively.  For example, the following all succeed:

     %!test assert (1+eps, 1, 2*eps)           # absolute error
     %!test assert (100+100*eps, 100, -2*eps)  # relative error

You can also do the comparison yourself, but still have assert generate the error:

     %!test assert (isempty ([]))
     %!test assert ([1, 2; 3, 4] > 0)

Because assert is so frequently used alone in a test block, there is a shorthand form:

     %!assert (...)

which is equivalent to:

     %!test assert (...)

Occasionally a block of tests will depend on having optional functionality in Octave.  Before testing such blocks the availability of the required functionality must be checked.  A %!testif HAVE_XXX block will only be run if Octave was compiled with functionality HAVE_XXX.  For example, the sparse single value decomposition, svds(), depends on having the ARPACK library.  All of the tests for svds begin with

     %!testif HAVE_ARPACK

Review config.h or __octave_config_info__ ("build_features") to see some of the possible values to check.

Sometimes during development there is a test that should work but is known to fail.  You still want to leave the test in because when the final code is ready the test should pass, but you may not be able to fix it immediately.  To avoid unnecessary bug reports for these known failures, mark the block with xtest rather than test:

     %!xtest assert (1==0)
     %!xtest fail ("success=1", "error")

In this case, the test will run and any failure will be reported. However, testing is not aborted and subsequent test blocks will be processed normally.  Another use of xtest is for statistical tests which should pass most of the time but are known to fail occasionally.

Each block is evaluated in its own function environment, which means that variables defined in one block are not automatically shared with other blocks.  If you do want to share variables, then you must declare them as shared before you use them.  For example, the following declares the variable A, gives it an initial value (default is empty), and then uses it in several subsequent tests.

     %!shared A
     %! A = [1, 2, 3; 4, 5, 6];
     %!assert (kron ([1; 2], A), [ A; 2*A ])
     %!assert (kron ([1, 2], A), [ A, 2*A ])
     %!assert (kron ([1,2; 3,4], A), [ A,2*A; 3*A,4*A ])

You can share several variables at the same time:

     %!shared A, B

Modifications to shared variables persist from one test to the next only if the test succeeds.  Thus, if one test modifies a shared variable, later tests cannot know which value of the shared variable to expect because the pass/fail status of earlier tests is unknown.  For this reason, it is not recommended to modify shared variables in tests.

You can also share test functions:

     %!function A = fn (B)
     %!  A = 2*B;
     %!endfunction
     %!assert (fn(2), 4)

Note that all previous variables and values are lost when a new shared block is declared.

Remember that %!function begins a new block and that %!endfunction ends this block.  Be aware that until a new block is started, lines starting with %!<space> will be discarded as comments. The following is nearly identical to the example above, but does nothing.

     %!function A = fn (B)
     %!  A = 2*B;
     %!endfunction
     %! assert (fn(2), 4)

Because there is a space after %! the assert statement does not begin a new block and this line is treated as a comment.

Error and warning blocks are like test blocks, but they only succeed if the code generates an error.  You can check the text of the error is correct using an optional regular expression <pattern>. For example:

     %!error <passes!> error ("this test passes!")

If the code doesn't generate an error, the test fails.  For example:

     %!error "this is an error because it succeeds."

produces

       ***** error "this is an error because it succeeds."
     !!!!! test failed: no error

It is important to automate the tests as much as possible, however some tests require user interaction.  These can be isolated into demo blocks, which if you are in batch mode, are only run when called with demo or the verbose option to test. The code is displayed before it is executed.  For example,

     %!demo
     %! T = [0:0.01:2*pi]; X = sin (T);
     %! plot (T, X);
     %! # you should now see a sine wave in your figure window

produces

     funcname example 1:
      T = [0:0.01:2*pi]; X = sin (T);
      plot (T, X);
      # you should now see a sine wave in your figure window

     Press <enter> to continue:

Note that demo blocks cannot use any shared variables.  This is so that they can be executed by themselves, ignoring all other tests.

If you want to temporarily disable a test block, put # in place of the block type.  This creates a comment block which is echoed in the log file but not executed.  For example:

     %!#demo
     %! T = [0:0.01:2*pi]; X = sin (T);
     %! plot (T, X);
     %! # you should now see a sine wave in your figure window

The following trivial code snippet provides examples for the use of fail, assert, error, and xtest:

     function OUTPUT = must_be_zero (INPUT)
       if (INPUT != 0)
         error ("Nonzero input!")
       endif
       OUTPUT = INPUT;
     endfunction

     %!fail ("must_be_zero (1)")
     %!assert (must_be_zero (0), 0)
     %!error <Nonzero> must_be_zero (1)
     %!xtest error ("This code generates an error")

When putting this in a file must_be_zero.m, and running the test, we see

     test must_be_zero verbose

     =>
     >>>>> /path/to/must_be_zero.m
     ***** fail ("must_be_zero (1)")
     ***** assert (must_be_zero (0), 0)
     ***** error <Nonzero> must_be_zero (1)
     ***** xtest error ("This code generates an error")
     !!!!! known failure
     This code generates an error
     PASSES 3 out of 4 tests (1 expected failure)

Block type summary:
...................

%!test
%!test <MESSAGE>
     Check that entire block is correct.  If <MESSAGE> is present, the test block is interpreted as for xtest.

%!testif HAVE_XXX
%!testif HAVE_XXX, HAVE_YYY, ...
%!testif HAVE_XXX, HAVE_YYY ...; RUNTIME_COND
%!testif ... <MESSAGE>
     Check block only if Octave was compiled with feature HAVE_XXX. RUNTIME_COND is an optional expression to evaluate to check whether some condition is met when the test is executed.  If RUNTIME_COND is false, the test is skipped.  If <MESSAGE> is present, the test block is interpreted as for xtest.

%!xtest
%!xtest <MESSAGE>
     Check block, report a test failure but do not abort testing.  If <MESSAGE> is present, then the text of the message is displayed if the test fails, like this:

          !!!!! known bug:  MESSAGE

     If the message is an integer, it is interpreted as a bug ID for the Octave bug tracker and reported as

          !!!!! known bug: https://octave.org/testfailure/?BUG-ID

     in which BUG-ID is the integer bug number.  The intent is to allow clearer documentation of known problems.

     If MESSAGE is an integer preceeded by an asterisk (e.g., *12345), it is interpreted as the id for a bug report that has been closed.  That usually means that the issue probed in this test has been resolved.  If such tests are failing, they are reported as regressions by the test function:

          !!!!! regression: https://octave.org/testfailure/?BUG-ID

%!error
%!error <MESSAGE>
%!warning
%!warning <MESSAGE>
     Check for correct error or warning message.  If <MESSAGE> is supplied it is interpreted as a regular expression pattern that is expected to match the error or warning message.

%!demo
     Demo only executes in interactive mode.

%!#
     Comment.  Ignore everything within the block

%!shared x,y,z
     Declare variables for use in multiple tests.

%!function
     Define a function for use in multiple tests.

%!endfunction
     Close a function definition.

%!assert (x, y, tol)

%!assert <MESSAGE> (x, y, tol)

%!fail (CODE, PATTERN)

%!fail <MESSAGE> (CODE, PATTERN)
     Shorthand for %!test assert (x, y, tol) or %!test fail (CODE, PATTERN).  If <MESSAGE> is present, the test block is interpreted as for xtest.

When coding tests the Octave convention is that lines that begin with a block type do not have a semicolon at the end.  Any code that is within a block, however, is normal Octave code and usually will have a trailing semicolon. For example,

     ## bare block instantiation
     %!assert (sin (0), 0)

but

     ## test block with normal Octave code
     %!test
     %! assert (sin (0), 0);

You can also create test scripts for built-in functions and your own C++ functions.  To do so, put a file with the bare function name (no .m extension) in a directory in the load path and it will be discovered by the test function.  Alternatively, you can embed tests directly in your C++ code:

     /*
     %!test disp ("this is a test")
     */

or

     #if 0
     %!test disp ("this is a test")
     #endif

However, in this case the raw source code will need to be on the load path and the user will have to remember to type test ("funcname.cc").

 -- assert: (COND)
 -- assert: (COND, ERRMSG)
 -- assert: (COND, ERRMSG, ...)
 -- assert: (COND, MSG_ID, ERRMSG, ...)
 -- assert: (OBSERVED, EXPECTED)
 -- assert: (OBSERVED, EXPECTED, TOL)

     Produce an error if the specified condition is not met.

     assert can be called in three different ways.

     assert (COND)
     assert (COND, ERRMSG)
     assert (COND, ERRMSG, ...)
     assert (COND, MSG_ID, ERRMSG, ...)
          Called with a single argument COND, assert produces an error if COND is false (numeric zero).

          Any additional arguments are passed to the error function for processing.

     assert (OBSERVED, EXPECTED)
          Produce an error if observed is not the same as expected.

          Note that OBSERVED and EXPECTED can be scalars, vectors, matrices, strings, cell arrays, or structures.

     assert (OBSERVED, EXPECTED, TOL)
          Produce an error if observed is not the same as expected but equality comparison for numeric data uses a tolerance TOL.

          If TOL is positive then it is an absolute tolerance which will produce an error if abs (OBSERVED - EXPECTED) > abs (TOL).

          If TOL is negative then it is a relative tolerance which will produce an error if abs (OBSERVED - EXPECTED) > abs (TOL * EXPECTED).

          If EXPECTED is zero TOL will always be interpreted as an absolute tolerance.

          If TOL is not scalar its dimensions must agree with those of OBSERVED and EXPECTED and tests are performed on an element-by-element basis.

     fail, test, error, isequal

 -- STATUS =: fail (CODE)
 -- STATUS =: fail (CODE, PATTERN)
 -- STATUS =: fail (CODE, "warning")
 -- STATUS =: fail (CODE, "warning", PATTERN)

     Return true if CODE fails with an error message matching PATTERN, otherwise produce an error.

     CODE must be in the form of a string that is passed to the Octave interpreter via the evalin function, i.e., a (quoted) string constant or a string variable.

     Note that if CODE runs successfully, rather than failing, the error printed is:

                    expected error <.> but got none

     If called with two arguments, the return value will be true only if CODE fails with an error message containing PATTERN (case sensitive).  If the code fails with a different error than the one specified in PATTERN then the message produced is:

                    expected <PATTERN>
                    but got <text of actual error>

     The angle brackets are not part of the output.

     When called with the "warning" option fail will produce an error if executing the code produces no warning. assert, error

Demonstration Functions
=======================

 -- demo: NAME
 -- demo: NAME N
 -- demo: ("NAME")
 -- demo: ("NAME", N)

     Run example code block N associated with the function NAME.

     If N is not specified, all examples are run.

     The preferred location for example code blocks is embedded within the script m-file immediately following the code that it exercises.  Alternatively, the examples may be stored in a file with the same name but no extension located on Octave's load path.  To separate examples from regular script code all lines are prefixed by %!.  Each example must also be introduced by the keyword "demo" flush left to the prefix with no intervening spaces.  The remainder of the example can contain arbitrary Octave code.  For example:

          %!demo
          %! t = 0:0.01:2*pi;
          %! x = sin (t);
          %! plot (t, x);
          %! title ("one cycle of a sine wave");
          %! #-------------------------------------------------
          %! # the figure window shows one cycle of a sine wave

     Note that the code is displayed before it is executed so that a simple comment at the end suffices for labeling what is being shown.  For plots, labeling can also be done with title or text.  It is generally not necessary to use disp or printf within the demo.

     Demos are run in a stand-alone function environment with no access to external variables.  This means that every demo must have separate initialization code.  Alternatively, all demos can be combined into a single large demo with the code

          %! input ("Press <enter> to continue: ", "s");

     between the sections, but this usage is discouraged.  Other techniques to avoid multiple initialization blocks include using multiple plots with a new figure command between each plot, or using subplot to put multiple plots in the same window.

     Finally, because demo evaluates within a function context it is not possible to define new functions within the code.  Anonymous functions make a good substitute in most instances.  If function blocks must be used then the code eval (example ("function", n)) will allow Octave to see them.  This has its own problems, however, as eval only evaluates one line or statement at a time.  In this case the function declaration must be wrapped with "if 1 <demo stuff> endif" where "if" is on the same line as "demo".  For example:

          %!demo if 1
          %!  function y = f(x)
          %!    y = x;
          %!  endfunction
          %!  f(3)
          %! endif

     rundemos, example, test

 -- example: NAME
 -- example: NAME N
 -- example: ("NAME")
 -- example: ("NAME", N)
 -- [CODESTR, CODEIDX] =: example (...)

     Display the code for example N associated with the function NAME, but do not run it.

     If N is not specified, all examples are displayed.

     When called with output arguments, the examples are returned in the form of a string CODESTR, with CODEIDX indicating the ending position of the various examples.

     For a complete explanation see demo. demo, test

 -- oruntests: ()
 -- oruntests: (DIRECTORY)
     Execute built-in tests for all m-files in the specified DIRECTORY.

     Test blocks in any C++ source files (*.cc) will also be executed for use with dynamically linked oct-file functions.

     If no directory is specified, operate on all directories in Octave's search path for functions. rundemos, test, path

 -- rundemos: ()
 -- rundemos: (DIRECTORY)
     Execute built-in demos for all m-files in the specified DIRECTORY.

     Demo blocks in any C++ source files (*.cc) will also be executed for use with dynamically linked oct-file functions.

     If no directory is specified, operate on all directories in Octave's search path for functions. demo, oruntests, path

 -- speed: (F, INIT, MAX_N, F2, TOL)
 -- [ORDER, N, T_F, T_F2] =: speed (...)

     Determine the execution time of an expression (F) for various input values (N).

     The N are log-spaced from 1 to MAX_N.  For each N, an initialization expression (INIT) is computed to create any data needed for the test.  If a second expression (F2) is given then the execution times of the two expressions are compared.  When called without output arguments the results are printed to stdout and displayed graphically.

     F
          The code expression to evaluate.

     MAX_N
          The maximum test length to run.  The default value is 100.  Alternatively, use [min_n, max_n] or specify the N exactly with [n1, n2, ..., nk].

     INIT
          Initialization expression for function argument values.  Use K for the test number and N for the size of the test.  This should compute values for all variables used by F.  Note that INIT will be evaluated first for k = 0, so things which are constant throughout the test series can be computed once.  The default value is X = randn (N, 1).

     F2
          An alternative expression to evaluate, so that the speed of two expressions can be directly compared.  The default is [].

     TOL
          Tolerance used to compare the results of expression F and expression F2.  If TOL is positive, the tolerance is an absolute one. If TOL is negative, the tolerance is a relative one.  The default is eps.  If TOL is Inf, then no comparison will be made.

     ORDER
          The time complexity of the expression O(a*n^p).  This is a structure with fields a and p.

     N
          The values N for which the expression was calculated AND the execution time was greater than zero.

     T_F
          The nonzero execution times recorded for the expression F in seconds.

     T_F2
          The nonzero execution times recorded for the expression F2 in seconds. If required, the mean time ratio is simply mean (T_f ./ T_f2).

     The slope of the execution time graph shows the approximate power of the asymptotic running time O(n^p).  This power is plotted for the region over which it is approximated (the latter half of the graph).  The estimated power is not very accurate, but should be sufficient to determine the general order of an algorithm.  It should indicate if, for example, the implementation is unexpectedly O(n^2) rather than O(n) because it extends a vector each time through the loop rather than pre-allocating storage.  In the current version of Octave, the following is not the expected O(n).

          speed ("for i = 1:n, y{i} = x(i); endfor", "", [1000, 10000])

     But it is if you preallocate the cell array y:

          speed ("for i = 1:n, y{i} = x(i); endfor", ...
                 "x = rand (n, 1); y = cell (size (x));", [1000, 10000])

     An attempt is made to approximate the cost of individual operations, but it is wildly inaccurate.  You can improve the stability somewhat by doing more work for each n.  For example:

          speed ("airy(x)", "x = rand (n, 10)", [10000, 100000])

     When comparing two different expressions (F, F2), the slope of the line on the speedup ratio graph should be larger than 1 if the new expression is faster.  Better algorithms have a shallow slope.  Generally, vectorizing an algorithm will not change the slope of the execution time graph, but will shift it relative to the original.  For example:

          speed ("sum (x)", "", [10000, 100000], ...
                 "v = 0; for i = 1:length (x), v += x(i); endfor")

     The following is a more complex example.  If there was an original version of xcorr using for loops and a second version using an FFT, then one could compare the run speed for various lags as follows, or for a fixed lag with varying vector lengths as follows:

          speed ("xcorr (x, n)", "x = rand (128, 1);", 100,
                 "xcorr_orig (x, n)", -100*eps)
          speed ("xcorr (x, 15)", "x = rand (20+n, 1);", 100,
                 "xcorr_orig (x, n)", -100*eps)

     Assuming one of the two versions is in xcorr_orig, this would compare their speed and their output values.  Note that the FFT version is not exact, so one must specify an acceptable tolerance on the comparison 100*eps. In this case, the comparison should be computed relatively, as abs ((X - Y) ./ Y) rather than absolutely as abs (X - Y).

     Type example ("speed") to see some real examples or demo ("speed") to run them.