Signature test needed checking compliance of implementations with JDO2.0 API

• Key: JDO-299
• Type: Task
• Status: Resolved
• Resolution: Fixed
• Priority: Minor
• Assignee: Martin Zaun
• Reporter: Martin Zaun
• Component/s: tck2
• Fix Version/s: JDO 2 final

Comments

[Martin Zaun]

This test uses reflection to verify that a set of JDO2.0 API classes conforms with the specification.

Since the number of features to be tested is quite large (~690 interfaces, classes, methods, or constants),
the feature definitions are not hard-coded into the test but parsed from a text file -- see attached file.

The format of the feature definition file is simple to parse, read, and maintain: basically, it's a stripped
version of the JDO2.0 Java interface and class sources.

[Craig Russell]

DataStoreCache ok.
JDOConnection ok.
Extent ok.
FetchPlan should verify the values of the static fields.
Do you intend to test the "implements" clause of the interfaces?
In general, I think we should not verify protected or private methods.
XXXIdentity ok
Reviewed up to but not including InstanceCallbacks.

[Martin Zaun]

attached is an updated version of the API signature descriptor file.

> FetchPlan should verify the values of the static fields.

I updated the descriptor file for these constant field values:

public interface javax.jdo.FetchPlan:
    public static final String DEFAULT = "default";
    public static final String ALL = "all";
    public static final int DETACH_UNLOAD_FIELDS = 2;
    public static final int DETACH_LOAD_FIELDS = 1;
    public static final int FETCH_SIZE_GREEDY = -1;
    public static final int FETCH_SIZE_OPTIMAL = 0;

public class javax.jdo.listener.InstanceLifecycleEvent:
    public static final int CREATE = 0;
    public static final int LOAD = 1;
    public static final int STORE = 2;
    public static final int CLEAR = 3;
    public static final int DELETE = 4;
    public static final int DIRTY = 5;
    public static final int DETACH = 6;
    public static final int ATTACH = 7;

public interface javax.jdo.Query:
    public static final String JDOQL = "javax.jdo.query.JDOQL";
    public static final String SQL = "javax.jdo.query.SQL";

public interface javax.jdo.spi.PersistenceCapable:
    public static final byte READ_WRITE_OK = 0;
    public static final byte LOAD_REQUIRED = 1;
    public static final byte READ_OK = -1;
    public static final byte CHECK_READ = 1;
    public static final byte MEDIATE_READ = 2;
    public static final byte CHECK_WRITE = 4;
    public static final byte MEDIATE_WRITE = 8;
    public static final byte SERIALIZABLE = 16;

There are 4 public static fields in JDOPermission for which I cannot (easily)
add constant definitions (not literals). But perhaps they don't belong into
the descriptor file anyway (they're not mentioned in the spec):

public final class javax.jdo.spi.JDOPermission:
    public static final javax.jdo.spi.JDOPermission GET_METADATA;
    public static final javax.jdo.spi.JDOPermission MANAGE_METADATA;
    public static final javax.jdo.spi.JDOPermission SET_STATE_MANAGER;
    public static final javax.jdo.spi.JDOPermission CLOSE_PERSISTENCE_MANAGER_FACTORY;

> Do you intend to test the "implements" clause of the interfaces?

I'm testing the 'extends' clause for interfaces (they cannot have 'implements',
which is also tested).

> In general, I think we should not verify protected or private methods.

I've removed all protected declarations from the descriptor file.

> Reviewed up to but not including InstanceCallbacks.

Thx, so far.

[Martin Zaun]

Attached is a fully testable version of the Signature Test for review/comments.

Except for the final integration with the TCK, I consider the version feature-complete:
- testing of superclasses, interfaces, parameters, exceptions, modifers
- testing public non-standard features found in the classes but not in the descriptor
- testing constant field values (static public final fields only, as in interfaces)
- support for (Java) user types name (opposed to reflection names) in the descriptor
- reports as many errors as possible (except for parse errors in the descriptor file)
- detailed reporting of missings, mismatches, loading problems
- reports summary statistics, current results:

Signature Test Results
======================

    tested features: 671

Successes:
    matching features: 671
    loadable classes: 91

Failures:
    missing features: 0
    non-matching features: 0
    non-standard features: 3
    unloadable classes: 0
    other problems: 0


Here's how to test. Save SignatureTest.java into
    tck20/src/java/org/apache/jdo/tck/api
(or let me know of a better place). Should build with the rest.

Invoke the test with 'java' by it's (nested) CLI class:
    org.apache.jdo.tck.api.SignatureTest\$CLI

The classpath needs to include
    tck20/target/classes
    api20/target/jdo2-api-SNAPSHOT.jar
    j2ee.jar

Without the j2ee.jar, there are failures reported on javax.jdo.Transaction,
which uses javax.transaction.Synchronization.

Provide as command-line argument a path to the downloaded signature descriptor
file:
    api20_defs.txt

Any progress/verbose output is send to stdout while any failures/problems are being
sent to stderr. So, you can save/look just the failure reports to a file: java ... 2> err.txt

Try changing the descriptor and check the result statistics and failure logs.

On the implementation, the code is organized into 5 classes, all in SignatureTest.java.
Not much redundancy, still ~1500 lines of code. if you have an idea on how to simplify,
let me know :)

Martin

[Craig Russell]

Looks good. A few comments.

1. The structure of the test classes is nonstandard. The classes in the file should either become members of the SignatureTest class or put into their own .java files.

2. The package structure of the classes needs to be improved. Should these classes all be put into org.apache.jdo.tck.util perhaps?

3. There should be a class that extends JDO_Test that internally calls the main method of the signature test. This could be a simple test class that adheres to the pattern for the tck tests. I've attached a sample test that you can use as a template.

4. The test needs to be integrated into the maven tck test framework. You will need to update the maven.xml and project.properties to implement a maven system property like "jdo.tck.signature.testdata" that is defined to be ${basedir}/src/conf/api20-signature.txt. This can then be retrieved by the JUnit test program.

5. The output should be split so only errors are shown. Maybe a -q quiet option and no output at all unless there is an error. Or a -v verbose option that prints everything. Need to have a way for the output to be returned to the calling program so it can be printed out as the test case exception message. This would be just the -q output.

6. When combining the verbose output and the error output, there is a threading problem. Maybe instead of using the system output and system error, you can use parameter arguments with file names that direct output to the logs directory.

[Craig Russell]

This is a simple class that fits into the jdo tck test framework and simply calls the CLI SignatureTest interface.

Also see the patch for some non-optimal but sample changes to JDO_Test, maven.xml.

[Martin Zaun]

Attached is a bundle signature_test_final_review.zip for review before checkin.
It contains the files:
   maven.xml.patch
   project.properties.patch
   src/conf/signatures.conf
   src/conf/jdo-2_0-signatures.txt
   src/java/org/apache/jdo/tck/api/SignatureTest.java
   src/java/org/apache/jdo/tck/util/signature/Formatter.java
   src/java/org/apache/jdo/tck/util/signature/SignatureVerifier.java
   src/java/org/apache/jdo/tck/util/signature/TypeHelper.java

Addressed are comments 1-6 from Craig:
- added integration with TCK
- refactored code into directories: ...tck/api and .../tck/utils/signature
- removed separate stderr stream but introduced options: -q and -v

A minor nuisance is that the test is run twice (datastore+application identity).

Currently the test failes with 3 public non-standard features reported.

[Craig Russell]

The patch contains the maven.xml patch twice. I assume that the project.properties actually sets the value of the signaturetest file?

The signature.conf has the wrong description.

Is there a way to only run the signature test once, since it's no value to run with both app identity and ds identitiy.

SignatureTest comment need not refer to JDO 2.0, as it's generic. When we update the spec to e.g. 2.0.1 or 2.1, we won't need to change this class, but just update the reference signature file.

You might define an additional SignatureTest constructor that has an explicit classloader argument. If not specified, the default is the classloader that you define here: classLoader = this.getClass().getClassLoader(). Then the comment "Tests the signature of classes (in the classpath) " would change to "Tests the signature of classes (in the specified classloader) ".

After you check this in, please file a JIRA on the three exceptions.

[Martin Zaun]

Updated the bundle for Craig's comments.

[Craig Russell]

Looks good. Go ahead and check it in.

[Martin Zaun]

Created issue JDO-341 for fixing the found discrepancies between: JDO2.0 spec v. API20 v. SignatureTest's descriptor file.

[Martin Zaun]

Committed the signature test with revision 386448.

[Martin Zaun]

I noticed that the SignatureTest does not get run as part of the maven runtck.jdori goal.

So, I've added "signatures.conf" to the 'jdo.tck.cfglist' property in src/conf/configurations.list.

An unresolved, remaining nuisance is that the test get's run twice (app/datastore identity), but
we don't know of any solution to this.

Unless there are any objections, I'm committing the following patch tomorrow.
Martin

$ svn diff src/conf/configurations.list
Index: src/conf/configurations.list
===================================================================
--- src/conf/configurations.list (revision 386483)
+++ src/conf/configurations.list (working copy)
@@ -26,4 +26,5 @@
                  inheritance4.conf \
                  schemaAttributeClass.conf \
                  schemaAttributeOrm.conf \
- schemaAttributePackage.conf
+ schemaAttributePackage.conf \
+ signatures.conf

[Craig Russell]

The maven.xml was updated so that tests in the runonce.conf are only run once per maven run, not once per identitytype and database. The configurations.list file was updated to include runonce.conf.

svn commit -m "JDO-299 Changed maven.xml to run signature test only once" maven.xml src/conf/runonce.conf src/conf/configurations.list src/conf/signatures.conf
Sending maven.xml
Sending src/conf/configurations.list
Adding src/conf/runonce.conf
Deleting src/conf/signatures.conf
Transmitting file data ...
Committed revision 386743.

[Craig Russell]

The maven.xml was updated so that tests in the runonce.conf are only run once per maven run, not once per identitytype and database. The configurations.list file was updated to include runonce.conf.

svn commit -m "JDO-299 Changed maven.xml to run signature test only once" maven.xml src/conf/runonce.conf src/conf/configurations.list src/conf/signatures.conf
Sending maven.xml
Sending src/conf/configurations.list
Adding src/conf/runonce.conf
Deleting src/conf/signatures.conf
Transmitting file data ...
Committed revision 386743.