runall should parse tests and examples in the same manner

• Key: STDCXX-261
• Type: Improvement
• Status: Open
• Priority: Minor
• Assignee: Unassigned
• Reporter: Andrew Black
• Component/s: Build and Installation
• Affects Version/s: 4.1.3, 4.2.0
• Fix Version/s: 4.3.0

Description

At this time, several of the self tests (0.cmdopts.out, 0.printf.out, 0.strncmp.out, 0.valcmp.out) produce output that lacks a summary section.

The runall.sh (or runall.c) utility should be able to compare the output of these tests against a reference file in the same manner that the output is compared for examples.

An extension of this concept is that it might make sense to unify the logic used to check the test and example results, so that the same process would be used for both tests and examples.

Comments

[Martin Sebor]

It would be nice if the runall aka exec utility could somehow figure out from the executable itself what kind of output to expect and adjust automatically. This would let it distinguish not only the tests designed to produce different output but also examples from tests, and it might open a whole bunch of other possibilities as well. One approach might be to embed some sort of a special string (e.g., in the form of an RCS comment) in the executable itself and having the utility read it (see http://www.delorie.com/gnu/docs/rcs/ident.1.html).

[Andrew Black]

The main drawback I can see with using RCS comments is that we would need to reimplement the ident utility in some manner to retrieve the RCS comments strings. ident isn't a part of the POSIX spec, and even if it were, it would be necessary to provide an implementation for windows.

A couple minor concerns would be ensuring that the RCS comment strings aren't optimized away and that RCS comments are inserted into the shell script (locale) targets.

To me, useful information that could be embedded would be the location of the reference input file (if one exists) and the reference output file. This would eliminate the search logic used at this time to locate said files. This would also allow tests to have such files. Another piece of useful information could be the format of the output file generated, allowing us to distinguish test suite driver output from compatibility layer output (eliminating the compat global and the -compat/-nocompat switches).

In an off-line discussion with Martin, he noted that another useful piece of information that could be embedded in a test would be platform-specific control over which tests are enabled/disabled. I wonder whether it would make more sense for such enable/disable logic to reside in the test suite/test driver, rather than the exec utility.

[Andrew Black]

Scheduling for 4.2.1, though it may be deferred again.

[Martin Sebor]

Added 4.2.0 to affected versions and deferred to 4.3.

[Martin Sebor]

We might want to consider this when extending the Expected Failure feature to handle failed test assertions since we'll most likely be changing the format of the test driver output.