Derby
  1. Derby
  2. DERBY-1758

Enable xmlSuite to run as part of derbyall in environments that have the required external jars.

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.2.1.6, 10.2.2.0, 10.3.1.4
    • Fix Version/s: 10.3.1.4
    • Component/s: Test
    • Labels:
      None
    • Urgency:
      Normal

      Description

      Due to the fact the XML support in Derby has external dependencies on Apache Xalan and a JAXP parser (such as, but not limited to, Apache Xerces), the tests for XML, which are all included in "xmlSuite", are not currently run as part of derbyall.

      Changes for DERBY-688 (and DERBY-567 indirectly) have now made it so that JVMs which have Apache Xalan and a JAXP parser embedded in them can run the XML tests without requiring additional jars, assuming that the embedded version of Xalan is at least the minimum version required by Derby (which is currently 2.5).

      So given that, the xmlSuite should be enabled as part of derbyall for any environment that is capable of running them.

      Per comments in DERBY-688, enabling the XML suite could include the following tasks:

      1. Enable the suite to run as part of derbyall but ONLY if the JVM has
      access to all of the required Xalan/JAXP classes.

      2. Resolve the following diff in lang/xmlBinding.java that occurs on
      some platforms (ex. Linux):

      < Inserted roughly 40k of data.
      10 del
      < Inserted roughly 40k of data.
      10a9,10
      > Inserted roughly 39k of data.
      > Inserted roughly 37k of data.
      21 del
      < 1, [ roughly 40k ]
      22 del
      < 2, [ roughly 40k ]
      22a21,22
      > 1, [ roughly 39k ]
      > 2, [ roughly 37k ]

      3. Add new tests to verify Derby behavior when XML classes are
      not present.

      Note that #3 may turn out to be its own Jira issue; the first two, however, should both be addressed as part of this issue since the xmlSuite will not run (and pass) on all platforms if either 1 or 2 is not addressed.

      1. d1758_newJUnitTests_v1.patch
        105 kB
        A B
      2. d1758_newJUnitTests_v1.stat
        0.4 kB
        A B
      3. d1758_newJUnitTests_v2.patch
        112 kB
        A B
      4. d1758_followup_v1.patch
        21 kB
        A B
      5. d1758_remove_xgen_v1.patch
        221 kB
        A B
      6. d1758_remove_xgen_v1.stat
        0.5 kB
        A B
      7. d1758_newXBindTest_v1.patch
        23 kB
        A B
      8. d1758_newSecMgr_doNotCommit.patch
        19 kB
        A B
      9. d1758_newXMLSuite_v1.patch
        8 kB
        A B
      10. d1758_remove_xbindAndSuite.patch
        32 kB
        A B
      11. d1758_secmgr_v1.patch
        13 kB
        A B
      12. d1758_secmgr_v1.stat
        0.5 kB
        A B

        Issue Links

          Activity

          Hide
          A B added a comment -

          Attaching a "phase 1" patch that adds two JUnit tests to lang/_Suite.java. The first test, XMLTypeAndOpsTest.java, is meant to be a JUnit equivalent to the current lang/xml_general.sql test. The second test, XMLMissingClassesTest, tests the behavior of the SQL/XML operators when the required JAXP or Xalan classes are not in the classpath.

          This patch constitutes my first attempt at writing JUnit tests, so any feedback/advice would be much appreciated. In particular, things I am wondering include:

          1. The expectation is that tests in XMLTypeAndOpsTest are run in
          the order in which they appear in the .java file. With all of
          my testing this seems to be the case--but as a general rule
          is this a safe assumption with JUnit tests? If not then I'll
          have to spend more time trying to make the different test cases
          independent of each other.

          2. Tables and data for the tests are created/inserted as they
          are needed. I think this is best for this particular test
          because the creation of tables and insertion of data is
          part of what we're testing--i.e. it's not just "fixture"
          code. But that said, I imagine some might prefer that
          all of the table creation and data insertion logic go
          into a startUp() method of a decorator that then runs
          the test. I don't know what the established protocol is
          with this. I prefer (at least in XMLTypeAndOpsTest) to
          do the creation of tables on the fly (esp. since that's
          what the old lang/xml_general.sql test did) but am not
          opposed to changing it.

          3. At most one of XMLTypeAndOpsTest and XMLMissingClasses will
          ever run in a given environment. Given that fact, I didn't
          necessarily have to split them out into two different tests--
          I could just as easily have added logic to the "suite()"
          method to add the appropriate test case(s) depending on
          whether or the classpath had the required classes. Is there
          a preference one way or the other for this kind of thing?
          (I'm impartial).

          The patch, d1758_newJUnitTests_v1.patch, also adds a new utility method and some associated state to JDBC.java for checking two things: 1) that the classpath has JAXP and Xalan classes, and 2) if the classpath has Xalan, check that the version of Xalan meets the minimum requirement for use of Derby SQL/XML operators. These methods/flags are then used to determine when to run the new XML JUnit tests. I wasn't sure if they are more appropriate in JDBC.java or BaseJDBCTestCase.java, so I chose the former. But if that was the wrong decision, I can of course change it.

          I ran the new test cases against jdk142, ibm142, jdk15, ibm15, and jdk16. In the absence of "endorsed jars" and classpath entries that include Xalan, the results were as follow:

          jdk142: Neither XMLTypeAndOpsTest nor XMLMissingClassesTest runs.
          This is because jdk142 exposes a version of Xalan that does
          not meet the minimum requirements for Derby SQL/XML. Attempts
          to run either test would lead to a Xalan-induced error caused
          by a Xalan bug when running with security manager. Thus both
          tests are skipped.

          ibm142: XMLTypeAndOpsTest runs successfully, XMLMissingClassesTest
          is skipped (as expected--a version of Xalan 2.6 is exposed in the
          ibm142 JVM).

          jdk15: XMLMissingClassesTest runs successfully, XMLTypeAndOpsTest
          is skipped (as expected--jdk15 does not expose Xalan).

          ibm15: XMLTypeAndOpsTest runs successfully, XMLMissingClassesTest
          is skipped (as expected--a version of Xalan 2.7.1 is exposed
          in the ibm15 JVM).

          jdk16: XMLMissingClassesTest runs successfully, XMLTypeAndOpsTest
          is skipped (as expected--jdk16 does not expose Xalan).

          I then re-ran the tests after providing external Xalan and JAXP jars via classpath settings and/or "endorsed jars" to the JVMs. I tried both a 2.5.0 and a 2.7.0 version of Xalan in these tests. In all cases the XMLMissingClassesTest was skipped (as expected) and the XMLTypeAndOpsTest was run. There were two cases where the test failed:

          1) ibm15 w/ endorsed 2.5.0: In this case there were lots of
          failures caused by Java exceptions saying something to the
          effect of "Unable to locate implementation for xxx" where
          "xxx" was an internal Xalan class. My guess is that since
          ibm15 embeds and exposes a version of Xalan 2.7.1 it has
          some dependencies on that particular version. When I
          overrode that with an earlier version (2.5.0) the classes
          expected by the JVM did not exist, leading to errors. That
          seems reasonable to me and is not something I think should
          block the patch. As a general rule it seems like a bad idea
          to override the JVMs libraries with earlier versions than
          what it expects.

          Note that when I endorsed Xalan 2.7.0 over the ibm15-embedded
          version, everything ran as expected.

          2) jdk16 w/ endorsed jars (either 2.5.0 or 2.7.0): there is
          a single failure reported in XMLTypeAndOpsTest when running
          against the Derby client. The failure occurs because the
          test is written to assert the SQLSTATE of a nested exception--
          but it turns out that for jdk16 the exception message does not
          include the SQLSTATE for the nested exception. Some changes
          that I made as part of DERBY-1976 added logic to assertSQLState()
          to search for nested exceptions, but that logic expects the
          SQLSTATE to be included in the error message. For whatever
          reason that's not the case with jdk16. More specifically,
          we see the following error message from the client with
          JVMs < 1.6:

          org.apache.derby.client.am.SqlException: The exception 'java.sql.SQLException:
          XML values are not allowed in top-level result sets; try using XMLSERIALIZE.'
          was thrown while evaluating an expression. SQLSTATE: 42Z71: XML values are not
          allowed in top-level result sets; try using XMLSERIALIZE.

          From this we can search for "SQLSTATE" and the expected "42Z71".
          But with jdk16 the error message is:

          java.sql.SQLException: The exception 'java.sql.SQLSyntaxErrorException: XML
          values are not allowed in top-level result sets; try using XMLSERIALIZE.' was
          thrown while evaluating an expression.

          The SQLSTATE of the nested exception is not printed. I spent some
          time trying to figure out how to retrieve the SQLSTATE of the nested
          exception in this case, but was unable to do so. I need to look
          at this some more.

          But since jdk16 by default does not exposed Xalan jars, the typical
          case for running tests on that JVM will be that XMLMissingClassesTest
          runs (and passes). Thus I don't think this is a blocker for the patch,
          either.

          And finally, I ran "suites.All" with jdk142 and ibm142 and all tests passed with this patch.

          NOTE: After this patch has been reviewed/updated and finally committed I will post a separate patch to remove the old lang/xml_general.sql test and the corresponding master files. I will then continue addressing the rest of the tasks for this issue (esp. xmlBinding.java) in incremental fashion.

          Show
          A B added a comment - Attaching a "phase 1" patch that adds two JUnit tests to lang/_Suite.java. The first test, XMLTypeAndOpsTest.java, is meant to be a JUnit equivalent to the current lang/xml_general.sql test. The second test, XMLMissingClassesTest, tests the behavior of the SQL/XML operators when the required JAXP or Xalan classes are not in the classpath. This patch constitutes my first attempt at writing JUnit tests, so any feedback/advice would be much appreciated. In particular, things I am wondering include: 1. The expectation is that tests in XMLTypeAndOpsTest are run in the order in which they appear in the .java file. With all of my testing this seems to be the case--but as a general rule is this a safe assumption with JUnit tests? If not then I'll have to spend more time trying to make the different test cases independent of each other. 2. Tables and data for the tests are created/inserted as they are needed. I think this is best for this particular test because the creation of tables and insertion of data is part of what we're testing--i.e. it's not just "fixture" code. But that said, I imagine some might prefer that all of the table creation and data insertion logic go into a startUp() method of a decorator that then runs the test. I don't know what the established protocol is with this. I prefer (at least in XMLTypeAndOpsTest) to do the creation of tables on the fly (esp. since that's what the old lang/xml_general.sql test did) but am not opposed to changing it. 3. At most one of XMLTypeAndOpsTest and XMLMissingClasses will ever run in a given environment. Given that fact, I didn't necessarily have to split them out into two different tests-- I could just as easily have added logic to the "suite()" method to add the appropriate test case(s) depending on whether or the classpath had the required classes. Is there a preference one way or the other for this kind of thing? (I'm impartial). The patch, d1758_newJUnitTests_v1.patch, also adds a new utility method and some associated state to JDBC.java for checking two things: 1) that the classpath has JAXP and Xalan classes, and 2) if the classpath has Xalan, check that the version of Xalan meets the minimum requirement for use of Derby SQL/XML operators. These methods/flags are then used to determine when to run the new XML JUnit tests. I wasn't sure if they are more appropriate in JDBC.java or BaseJDBCTestCase.java, so I chose the former. But if that was the wrong decision, I can of course change it. I ran the new test cases against jdk142, ibm142, jdk15, ibm15, and jdk16. In the absence of "endorsed jars" and classpath entries that include Xalan, the results were as follow: jdk142: Neither XMLTypeAndOpsTest nor XMLMissingClassesTest runs. This is because jdk142 exposes a version of Xalan that does not meet the minimum requirements for Derby SQL/XML. Attempts to run either test would lead to a Xalan-induced error caused by a Xalan bug when running with security manager. Thus both tests are skipped. ibm142: XMLTypeAndOpsTest runs successfully, XMLMissingClassesTest is skipped (as expected--a version of Xalan 2.6 is exposed in the ibm142 JVM). jdk15: XMLMissingClassesTest runs successfully, XMLTypeAndOpsTest is skipped (as expected--jdk15 does not expose Xalan). ibm15: XMLTypeAndOpsTest runs successfully, XMLMissingClassesTest is skipped (as expected--a version of Xalan 2.7.1 is exposed in the ibm15 JVM). jdk16: XMLMissingClassesTest runs successfully, XMLTypeAndOpsTest is skipped (as expected--jdk16 does not expose Xalan). I then re-ran the tests after providing external Xalan and JAXP jars via classpath settings and/or "endorsed jars" to the JVMs. I tried both a 2.5.0 and a 2.7.0 version of Xalan in these tests. In all cases the XMLMissingClassesTest was skipped (as expected) and the XMLTypeAndOpsTest was run. There were two cases where the test failed: 1) ibm15 w/ endorsed 2.5.0: In this case there were lots of failures caused by Java exceptions saying something to the effect of "Unable to locate implementation for xxx" where "xxx" was an internal Xalan class. My guess is that since ibm15 embeds and exposes a version of Xalan 2.7.1 it has some dependencies on that particular version. When I overrode that with an earlier version (2.5.0) the classes expected by the JVM did not exist, leading to errors. That seems reasonable to me and is not something I think should block the patch. As a general rule it seems like a bad idea to override the JVMs libraries with earlier versions than what it expects. Note that when I endorsed Xalan 2.7.0 over the ibm15-embedded version, everything ran as expected. 2) jdk16 w/ endorsed jars (either 2.5.0 or 2.7.0): there is a single failure reported in XMLTypeAndOpsTest when running against the Derby client. The failure occurs because the test is written to assert the SQLSTATE of a nested exception-- but it turns out that for jdk16 the exception message does not include the SQLSTATE for the nested exception. Some changes that I made as part of DERBY-1976 added logic to assertSQLState() to search for nested exceptions, but that logic expects the SQLSTATE to be included in the error message. For whatever reason that's not the case with jdk16. More specifically, we see the following error message from the client with JVMs < 1.6: org.apache.derby.client.am.SqlException: The exception 'java.sql.SQLException: XML values are not allowed in top-level result sets; try using XMLSERIALIZE.' was thrown while evaluating an expression. SQLSTATE: 42Z71: XML values are not allowed in top-level result sets; try using XMLSERIALIZE. From this we can search for "SQLSTATE" and the expected "42Z71". But with jdk16 the error message is: java.sql.SQLException: The exception 'java.sql.SQLSyntaxErrorException: XML values are not allowed in top-level result sets; try using XMLSERIALIZE.' was thrown while evaluating an expression. The SQLSTATE of the nested exception is not printed. I spent some time trying to figure out how to retrieve the SQLSTATE of the nested exception in this case, but was unable to do so. I need to look at this some more. But since jdk16 by default does not exposed Xalan jars, the typical case for running tests on that JVM will be that XMLMissingClassesTest runs (and passes). Thus I don't think this is a blocker for the patch, either. And finally, I ran "suites.All" with jdk142 and ibm142 and all tests passed with this patch. NOTE: After this patch has been reviewed/updated and finally committed I will post a separate patch to remove the old lang/xml_general.sql test and the corresponding master files. I will then continue addressing the rest of the tasks for this issue (esp. xmlBinding.java) in incremental fashion.
          Hide
          A B added a comment -

          Unchecking "Patch Available" as I realized that I was confused about some setUp() and tearDown() functionality and that confusion makes the patch a bit awkward. Will post another patch after cleaning things up a bit...

          Show
          A B added a comment - Unchecking "Patch Available" as I realized that I was confused about some setUp() and tearDown() functionality and that confusion makes the patch a bit awkward. Will post another patch after cleaning things up a bit...
          Hide
          Daniel John Debrunner added a comment -

          Tests are not run in any guaranteed order if the set of tests is created using the class, e.g. new TestSuite(MyTest.class).

          See the second page of:

          http://www.javaworld.com/jw-12-2000/jw-1221-junit.html

          Show
          Daniel John Debrunner added a comment - Tests are not run in any guaranteed order if the set of tests is created using the class, e.g. new TestSuite(MyTest.class). See the second page of: http://www.javaworld.com/jw-12-2000/jw-1221-junit.html
          Hide
          Daniel John Debrunner added a comment -

          Maybe your new changes will address this, but I don't see why you declare fields to hold the various statement objects. Why not just use local variables for them? The fields will lead to increased memory usage for the tests as each test fixture will have its own copy of all the fields. Local scoping such items as local variables I believe makes the code much clearer, see where & how the variable is created inline with the test.

          Show
          Daniel John Debrunner added a comment - Maybe your new changes will address this, but I don't see why you declare fields to hold the various statement objects. Why not just use local variables for them? The fields will lead to increased memory usage for the tests as each test fixture will have its own copy of all the fields. Local scoping such items as local variables I believe makes the code much clearer, see where & how the variable is created inline with the test.
          Hide
          A B added a comment -

          > Tests are not run in any guaranteed order if the set of tests is created using the class, e.g. new TestSuite(MyTest.class).

          Thank you for pointing this out. I just found a quote from the JUnit documentation that says the same thing (http://junit.sourceforge.net/doc/faq/faq.htm#tests_2)/

          > http://www.javaworld.com/jw-12-2000/jw-1221-junit.html

          Great link, thanks for sharing it! I'm working to apply these suggestions as I prepare the second patch.

          > Maybe your new changes will address this, but I don't see why you declare fields to hold the various statement objects.

          Side effect of incremental development on my machine. The first thing I did was create the JUnit equivalent to xml_general.sql as a single test case, in which case object fields seemed like an okay thing to do. When I broke it up into a bunch of smaller test cases I didn't remove the object fields, hence they're still there.

          > The fields will lead to increased memory usage for the tests as each test fixture will have its own copy of all the fields.

          Interesting, I guess I didn't realize multiple test fixtures exist in memory at the same time. Somehow when I read "JUnit provides a new instance of the fixture objects for each test method" I was thinking that only one instance of the fixture is instantiated at a time. Thanks for pointing this out, I will make the fields local to each test method.

          And thanks for taking the time to review the changes. I appreciate the feedback!

          Show
          A B added a comment - > Tests are not run in any guaranteed order if the set of tests is created using the class, e.g. new TestSuite(MyTest.class). Thank you for pointing this out. I just found a quote from the JUnit documentation that says the same thing ( http://junit.sourceforge.net/doc/faq/faq.htm#tests_2)/ > http://www.javaworld.com/jw-12-2000/jw-1221-junit.html Great link, thanks for sharing it! I'm working to apply these suggestions as I prepare the second patch. > Maybe your new changes will address this, but I don't see why you declare fields to hold the various statement objects. Side effect of incremental development on my machine. The first thing I did was create the JUnit equivalent to xml_general.sql as a single test case, in which case object fields seemed like an okay thing to do. When I broke it up into a bunch of smaller test cases I didn't remove the object fields, hence they're still there. > The fields will lead to increased memory usage for the tests as each test fixture will have its own copy of all the fields. Interesting, I guess I didn't realize multiple test fixtures exist in memory at the same time. Somehow when I read "JUnit provides a new instance of the fixture objects for each test method" I was thinking that only one instance of the fixture is instantiated at a time. Thanks for pointing this out, I will make the fields local to each test method. And thanks for taking the time to review the changes. I appreciate the feedback!
          Hide
          A B added a comment -

          Posting a second version of the new JUnit tests, d1758_newJUnitTests_v2.patch, that has the following changes w.r.t _v1:

          • Adds a new TestSetup class that creates and populates two tables to
            be used as a fixture for the various test methods in XMLTypeAndOpsTest.
            These two tables are used in, and constant across, several different
            test methods. Use of a new TestSetup class allows the tables to be
            created once for the entire test, instead of once-per-test-method.
          • Moves the creation/cleanup of all other tables to the local test methods
            in which they are used.
          • Makes all test methods independent of each other so that the order
            in which the tests are executed does not matter. Also ensures that
            no test method has "side effects" on the test fixture. These
            changes are per JUnit recommendations noted by Dan in his review
            comments.
          • Moves creation of local fields (such as statement objects) into the
            local test methods in which they are used (per Dan's review comments).
          • Adds a check of "JDBC.vmSupportsJDBC4()" to the appropriate test
            method so that the test will pass for jdk16 (see previous comment
            regarding SQLSTATEs and nested exceptions in jdk16).

          I ran the new JUnit tests against ibm142, jdk142, ibm15, jdk15, and jdk16 and it passed in all cases.

          If anyone is inclined to further review this patch, I'd be grateful for any other suggestions...

          NOTE: "svn stat" for the _v2 patch is the same as for the _v1 patch.

          Show
          A B added a comment - Posting a second version of the new JUnit tests, d1758_newJUnitTests_v2.patch, that has the following changes w.r.t _v1: Adds a new TestSetup class that creates and populates two tables to be used as a fixture for the various test methods in XMLTypeAndOpsTest. These two tables are used in, and constant across, several different test methods. Use of a new TestSetup class allows the tables to be created once for the entire test, instead of once-per-test-method. Moves the creation/cleanup of all other tables to the local test methods in which they are used. Makes all test methods independent of each other so that the order in which the tests are executed does not matter. Also ensures that no test method has "side effects" on the test fixture. These changes are per JUnit recommendations noted by Dan in his review comments. Moves creation of local fields (such as statement objects) into the local test methods in which they are used (per Dan's review comments). Adds a check of "JDBC.vmSupportsJDBC4()" to the appropriate test method so that the test will pass for jdk16 (see previous comment regarding SQLSTATEs and nested exceptions in jdk16). I ran the new JUnit tests against ibm142, jdk142, ibm15, jdk15, and jdk16 and it passed in all cases. If anyone is inclined to further review this patch, I'd be grateful for any other suggestions... NOTE: "svn stat" for the _v2 patch is the same as for the _v1 patch.
          Hide
          Olav Sandstaa added a comment -

          > 2) jdk16 w/ endorsed jars (either 2.5.0 or 2.7.0): there is
          > a single failure reported in XMLTypeAndOpsTest when running
          > against the Derby client. The failure occurs because the
          > test is written to assert the SQLSTATE of a nested exception--
          > but it turns out that for jdk16 the exception message does not
          > include the SQLSTATE for the nested exception. Some changes
          > that I made as part of DERBY-1976 added logic to assertSQLState()
          > to search for nested exceptions, but that logic expects the
          > SQLSTATE to be included in the error message. For whatever
          > reason that's not the case with jdk16. More specifically,
          > we see the following error message from the client with
          > JVMs < 1.6:
          >
          > org.apache.derby.client.am.SqlException: The exception 'java.sql.SQLException:
          > XML values are not allowed in top-level result sets; try using XMLSERIALIZE.'
          > was thrown while evaluating an expression. SQLSTATE: 42Z71: XML values are not
          > allowed in top-level result sets; try using XMLSERIALIZE.
          >
          > From this we can search for "SQLSTATE" and the expected "42Z71".
          > But with jdk16 the error message is:
          >
          > java.sql.SQLException: The exception 'java.sql.SQLSyntaxErrorException: XML
          > values are not allowed in top-level result sets; try using XMLSERIALIZE.' was
          > thrown while evaluating an expression.
          >
          > The SQLSTATE of the nested exception is not printed. I spent some
          > time trying to figure out how to retrieve the SQLSTATE of the nested
          > exception in this case, but was unable to do so. I need to look
          > at this some more.

          Was this with the embedded or the client driver? Based on the name of one of the execptions it seems that you were using the client driver at least in some of your experiments. If it was with the client driver there is a bug report for this (see DERBY-1440).

          Show
          Olav Sandstaa added a comment - > 2) jdk16 w/ endorsed jars (either 2.5.0 or 2.7.0): there is > a single failure reported in XMLTypeAndOpsTest when running > against the Derby client. The failure occurs because the > test is written to assert the SQLSTATE of a nested exception-- > but it turns out that for jdk16 the exception message does not > include the SQLSTATE for the nested exception. Some changes > that I made as part of DERBY-1976 added logic to assertSQLState() > to search for nested exceptions, but that logic expects the > SQLSTATE to be included in the error message. For whatever > reason that's not the case with jdk16. More specifically, > we see the following error message from the client with > JVMs < 1.6: > > org.apache.derby.client.am.SqlException: The exception 'java.sql.SQLException: > XML values are not allowed in top-level result sets; try using XMLSERIALIZE.' > was thrown while evaluating an expression. SQLSTATE: 42Z71: XML values are not > allowed in top-level result sets; try using XMLSERIALIZE. > > From this we can search for "SQLSTATE" and the expected "42Z71". > But with jdk16 the error message is: > > java.sql.SQLException: The exception 'java.sql.SQLSyntaxErrorException: XML > values are not allowed in top-level result sets; try using XMLSERIALIZE.' was > thrown while evaluating an expression. > > The SQLSTATE of the nested exception is not printed. I spent some > time trying to figure out how to retrieve the SQLSTATE of the nested > exception in this case, but was unable to do so. I need to look > at this some more. Was this with the embedded or the client driver? Based on the name of one of the execptions it seems that you were using the client driver at least in some of your experiments. If it was with the client driver there is a bug report for this (see DERBY-1440 ).
          Hide
          A B added a comment -

          > Was this with the embedded or the client driver?

          Yes you are correct, this is with the client driver only. Sorry for not making that more clear.

          > If it was with the client driver there is a bug report for this (see DERBY-1440).

          Ah, excellent! I did a search fo Jira but somehow managed to miss this one. Thank you very much for tracking it down.

          Show
          A B added a comment - > Was this with the embedded or the client driver? Yes you are correct, this is with the client driver only. Sorry for not making that more clear. > If it was with the client driver there is a bug report for this (see DERBY-1440 ). Ah, excellent! I did a search fo Jira but somehow managed to miss this one. Thank you very much for tracking it down.
          Hide
          Bryan Pendleton added a comment -

          I was successful with this patch. I was able to run the tests in several environments,
          including mixtures of JDK 1.4 and 1.5, with and without Xalan 2.7 as an endorsed library.

          The tests functioned as expected: when I had the right environment, the tests passed,
          and when I didn't ,the tests quietly did nothing.

          It was a little odd that, when the tests detected that I didn't have the requisite environment,
          and quietly exited without doing any work, the result from JUnit was simply:

          -bash-2.05b$ java junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest

          Time: 0

          OK (0 tests)

          If I was a Derby developer trying to get my environment set up to be able to run these tests,
          and I was having trouble getting my environment set up correctly, this output would not
          give me much to go on in terms of fixing my environment.

          But I guess that's not really the point of these tests.

          I didn't read every line of the patch (it's quite large), but in general I'm comfortable with it.

          Show
          Bryan Pendleton added a comment - I was successful with this patch. I was able to run the tests in several environments, including mixtures of JDK 1.4 and 1.5, with and without Xalan 2.7 as an endorsed library. The tests functioned as expected: when I had the right environment, the tests passed, and when I didn't ,the tests quietly did nothing. It was a little odd that, when the tests detected that I didn't have the requisite environment, and quietly exited without doing any work, the result from JUnit was simply: -bash-2.05b$ java junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest Time: 0 OK (0 tests) If I was a Derby developer trying to get my environment set up to be able to run these tests, and I was having trouble getting my environment set up correctly, this output would not give me much to go on in terms of fixing my environment. But I guess that's not really the point of these tests. I didn't read every line of the patch (it's quite large), but in general I'm comfortable with it.
          Hide
          Bryan Pendleton added a comment -

          Patch d1758_newJUnitTests_v2.patch committed to subversion as revision 468503.

          Cleared patch available flag.

          Army, there were two or three other submissions to JDBC.java after you attached your
          patch and before I committed. The merging seemed clean, but can you please check
          the results carefully and make sure that JDBC.java looks OK?

          Show
          Bryan Pendleton added a comment - Patch d1758_newJUnitTests_v2.patch committed to subversion as revision 468503. Cleared patch available flag. Army, there were two or three other submissions to JDBC.java after you attached your patch and before I committed. The merging seemed clean, but can you please check the results carefully and make sure that JDBC.java looks OK?
          Hide
          Daniel John Debrunner added a comment -

          Having seen the commit messages go by I would have to say that JDBC is not the right class for methods related to Xalan.
          The existing methods are there because they directly related to JDBC, Xalan does not.

          Maybe a new XML class?

          Show
          Daniel John Debrunner added a comment - Having seen the commit messages go by I would have to say that JDBC is not the right class for methods related to Xalan. The existing methods are there because they directly related to JDBC, Xalan does not. Maybe a new XML class?
          Hide
          A B added a comment -

          Thank you for the commit, Bryan!

          I'm attaching another patch, d1758_followup_v1.patch, that moves the XML utility methods out of junit.JDBC and into a new class, junit.XML, per Dan's suggestion (thanks Dan).

          Note that I changed the "haveClass()" method in JDBC.java from private to protected so that it can be called from the junit.XML class. That was the easiest thing to do.

          Since checking the classpath is not a JDBC-specific operation, the other option is to move "haveClass()" to some other class in the junit package. If anyone indicates a preference for doing so and also indicates the class to which the method should be moved, I can do it this way. Otherwise I'll just leave it as it is (i.e. keep it in JDBC.java and make it protected).

          I ran "suites.All" with this patch applied using jdk142 with endorsed Xalan 2.5 jars and there was one error--an OOM error with testLongLobLength(). That was probably because I was running the suite in the background while running other apps. In any event, I don't think that's related to this change.

          As always, review comments and/or commit are appreciated.

          Show
          A B added a comment - Thank you for the commit, Bryan! I'm attaching another patch, d1758_followup_v1.patch, that moves the XML utility methods out of junit.JDBC and into a new class, junit.XML, per Dan's suggestion (thanks Dan). Note that I changed the "haveClass()" method in JDBC.java from private to protected so that it can be called from the junit.XML class. That was the easiest thing to do. Since checking the classpath is not a JDBC-specific operation, the other option is to move "haveClass()" to some other class in the junit package. If anyone indicates a preference for doing so and also indicates the class to which the method should be moved, I can do it this way. Otherwise I'll just leave it as it is (i.e. keep it in JDBC.java and make it protected). I ran "suites.All" with this patch applied using jdk142 with endorsed Xalan 2.5 jars and there was one error--an OOM error with testLongLobLength(). That was probably because I was running the suite in the background while running other apps. In any event, I don't think that's related to this change. As always, review comments and/or commit are appreciated.
          Hide
          Bryan Pendleton added a comment -

          I think Dan's idea of a separate XML utility class makes sense, too. I think it's reasonable to
          leave the haveClass method in the JDBC class and call it from the XML class.

          I applied the follow-up patch and I successfully ran various combinations of 1.4 and 1.5 JDK
          with and without the Xalan classes available. Everything seemed to be working correctly to
          me with the XML Junit tests, so I committed d1758_followup_v1.patch to subversion
          as revision 468605.

          Show
          Bryan Pendleton added a comment - I think Dan's idea of a separate XML utility class makes sense, too. I think it's reasonable to leave the haveClass method in the JDBC class and call it from the XML class. I applied the follow-up patch and I successfully ran various combinations of 1.4 and 1.5 JDK with and without the Xalan classes available. Everything seemed to be working correctly to me with the XML Junit tests, so I committed d1758_followup_v1.patch to subversion as revision 468605.
          Hide
          A B added a comment -

          Thank you very much for the review and commit, Bryan!

          Show
          A B added a comment - Thank you very much for the review and commit, Bryan!
          Hide
          A B added a comment -

          Attaching a patch, d1758_remove_xgen_v1.patch, to remove the old lang/xml_general.sql test and the corresponding master files. That test has now been replaced by lang/XMLTypeAndOpsTest.java, which is a JUnit test.

          After this patch is committed, the remaining work for this issue will be to convert lang/xmlBinding.java to JUnit and ensure that it passes on all platforms. I will be working on that next...

          In the meantime, if anyone has time to review/commit the d1758_remove_xgen_v1.patch, I welcome the feedback.

          Show
          A B added a comment - Attaching a patch, d1758_remove_xgen_v1.patch, to remove the old lang/xml_general.sql test and the corresponding master files. That test has now been replaced by lang/XMLTypeAndOpsTest.java, which is a JUnit test. After this patch is committed, the remaining work for this issue will be to convert lang/xmlBinding.java to JUnit and ensure that it passes on all platforms. I will be working on that next... In the meantime, if anyone has time to review/commit the d1758_remove_xgen_v1.patch, I welcome the feedback.
          Hide
          Daniel John Debrunner added a comment -

          d1758_remove_xgen_v1.patch, applied - Thanks AB.

          Show
          Daniel John Debrunner added a comment - d1758_remove_xgen_v1.patch, applied - Thanks AB.
          Hide
          Daniel John Debrunner added a comment -

          The re-worked test XMLTypeAndOpsTest looks much better, I was think the improvements have made it so the suite() method could be simplified to

          return new XMLTestSetup(TestConfiguration.defaultSuite(XMLTypeAndOpsTest.class))

          but I realized that the CleanDatabasetestSetup installed by defaultSuite would negate the XMLTestSetup.

          TestConfiguration.defaultSuite() should probably be enhanced to have an overloaded signature with a boolean
          that allows optional addition of the CleanDatabasetestSetup.

          Show
          Daniel John Debrunner added a comment - The re-worked test XMLTypeAndOpsTest looks much better, I was think the improvements have made it so the suite() method could be simplified to return new XMLTestSetup(TestConfiguration.defaultSuite(XMLTypeAndOpsTest.class)) but I realized that the CleanDatabasetestSetup installed by defaultSuite would negate the XMLTestSetup. TestConfiguration.defaultSuite() should probably be enhanced to have an overloaded signature with a boolean that allows optional addition of the CleanDatabasetestSetup.
          Hide
          A B added a comment -

          Attaching a patch, d1758_newXBindTest_v1.patch, that adds a new JUnit test to replace the old lang/xmlBinding.java test. The patch does the following:

          • Adds XML file insertion utility methods to junit/XML.java
          • Creates a new JUnit test called lang/XMLBindingTest.java that
            uses the new utility methods to test various binding scenarios
            with Derby's SQL/XML operators.
          • Overloads the TestConfiguration.defaultSuite() method with a boolean
            signature to allow optional addition of CleanDatabaseSetup, per
            Dan's previous comment.
          • Updates lang/XMLTypeAndOpsTest to use the new overloaded defaultSuite()
            method.

          Note that I have not yet added XMLBindingTest to any suites because the test fails on Linux machines with a similar diff to what was seen for the old lang/xmlBinding.java test. I'm still looking into that failure and hope to post an appropriate follow-up patch in the next couple of days.

          I also plan to post subsequent patches to remove the old lang/xmlBinding.java test and to create a singe JUnit "suite" to run all of the XML tests. Those patches will come after the new XML binding JUnit test is working as expected on all platforms.

          I ran suites.All against classes (vs jars) with ibm142 (textui) as a sanity check and all tests passed. When I tried to run the suite with jars I kept seeing a Java core dump in the LangScripts.derived() test with ibm142. But that occurs even without my changes, so I'm not sure what's going on there. I also ran suites.All with Sun jdk142 (swingui): in that case there was no core dump and the suite ran with one failure: an OutOfMemory error in BlobClob4Blob.testPositionAgressive(). Again, I don't think my changes are the cause.

          With the exception of the TestConfiguration changes the patch is fairly isolated and should not affect suites.All (yet). If anyone has a chance to review, I'd appreciate it.

          In the absence of feedback/suggestions, I'll probably commit the patch tomorrow (Friday)...

          Show
          A B added a comment - Attaching a patch, d1758_newXBindTest_v1.patch, that adds a new JUnit test to replace the old lang/xmlBinding.java test. The patch does the following: Adds XML file insertion utility methods to junit/XML.java Creates a new JUnit test called lang/XMLBindingTest.java that uses the new utility methods to test various binding scenarios with Derby's SQL/XML operators. Overloads the TestConfiguration.defaultSuite() method with a boolean signature to allow optional addition of CleanDatabaseSetup, per Dan's previous comment. Updates lang/XMLTypeAndOpsTest to use the new overloaded defaultSuite() method. Note that I have not yet added XMLBindingTest to any suites because the test fails on Linux machines with a similar diff to what was seen for the old lang/xmlBinding.java test. I'm still looking into that failure and hope to post an appropriate follow-up patch in the next couple of days. I also plan to post subsequent patches to remove the old lang/xmlBinding.java test and to create a singe JUnit "suite" to run all of the XML tests. Those patches will come after the new XML binding JUnit test is working as expected on all platforms. I ran suites.All against classes (vs jars) with ibm142 (textui) as a sanity check and all tests passed. When I tried to run the suite with jars I kept seeing a Java core dump in the LangScripts.derived() test with ibm142. But that occurs even without my changes, so I'm not sure what's going on there. I also ran suites.All with Sun jdk142 (swingui): in that case there was no core dump and the suite ran with one failure: an OutOfMemory error in BlobClob4Blob.testPositionAgressive(). Again, I don't think my changes are the cause. With the exception of the TestConfiguration changes the patch is fairly isolated and should not affect suites.All (yet). If anyone has a chance to review, I'd appreciate it. In the absence of feedback/suggestions, I'll probably commit the patch tomorrow (Friday)...
          Hide
          A B added a comment -

          While trying to add lang/XMLBindingTest.java to a JUnit suite so that it can run as part of suites.All, I ran into a problem where the test setup fails because the JAXP parser does not have permissions to read the DTD file that is used for inserting test data.

          Note that this isn't a problem when running the test alone because when it runs standalone, the "setup" work happens before the security manager is installed. When the test is run as part of a suite, though, a security manager is installed for any tests that run prior to XMLBindingTest, and that security manager is not "uninstalled" after those tests. Then, since the lingering security manager does not give the necessary permissions to JAXP, the binding test fails during setup.

          This security problem was also an issue for the old binding test, lang/xmlBinding.java. In that case the problem was solved by creating a policy file with specific permissions to allow "everything" to read the DTD file. That permission was then automatically "merged" into the security manager's policy as part of the old harness setup, thereby allowing the test to run without error.

          So I made some changes to the JUnit security manager setup to allow a test to append permissions to a policy file and to then install a security manager with those permissions. However, when I try to run XMLBindingTest.java as part of a JUnit suite, the new permissions are never picked up.

          I did some tracing in the JUnit setup classes and I can verify that the code to "uninstall" a security manager and install a new manager that uses the modified policy is in fact being executed. However, it looks like the new SecurityManager() is never actually being enforced. Or put another way, it looks like the security policy that takes effect is always the first one to be assigned in the JVM; subsequent attempts to set the System security manager to non-null values will re-enable the security manager, but will not alter its policy.

          Note, though, that attempts to UN-install the security manager (by calling System.setSecurityManager(null)) do work. But when we try try to call setSecurityManager() with a non-null value (namely, "new SecurityManager()"), the policy that it implements is always the first policy that was successfully applied for the JVM; if the "java.security.policy" property is pointing to some other (new) policy file, it is ignored.

          At least, this is what I've come to conclude based on my understanding of the security manager setup code.

          In order to get the binding test to work as part of a suite I can just uninstall the security manager altogether-that works fine. However, that seems a bit like cheating to me. And to be honest, I think that we should be able to change the security manager to use a new policy-or at least, that has been my impression based on the code comments.

          So my question, then, is this: does anyone know if I have to do something special to get the test to install a Security Manager that implements a different policy file than one that has previously been installed? I've spent a lot of time on this but have come up empty-handed.

          I'm attaching a patch, d1758_newSecMgr_doNotCommit.patch with the test setup changes that I've made. If there are any problems (glaring or otherwise) with my approach that might be leading to this inability to alter security policy, I would be grateful to anyone who could point them out...

          Note the patch includes all of the changes that I've made to add XMLBindingTest.java to a new XML JUnit suite. I'm hoping anyone who looks at it can just ignore the test-specific changes and focus on the security-related ones...

          Show
          A B added a comment - While trying to add lang/XMLBindingTest.java to a JUnit suite so that it can run as part of suites.All, I ran into a problem where the test setup fails because the JAXP parser does not have permissions to read the DTD file that is used for inserting test data. Note that this isn't a problem when running the test alone because when it runs standalone, the "setup" work happens before the security manager is installed. When the test is run as part of a suite, though, a security manager is installed for any tests that run prior to XMLBindingTest, and that security manager is not "uninstalled" after those tests. Then, since the lingering security manager does not give the necessary permissions to JAXP, the binding test fails during setup. This security problem was also an issue for the old binding test, lang/xmlBinding.java. In that case the problem was solved by creating a policy file with specific permissions to allow "everything" to read the DTD file. That permission was then automatically "merged" into the security manager's policy as part of the old harness setup, thereby allowing the test to run without error. So I made some changes to the JUnit security manager setup to allow a test to append permissions to a policy file and to then install a security manager with those permissions. However, when I try to run XMLBindingTest.java as part of a JUnit suite, the new permissions are never picked up. I did some tracing in the JUnit setup classes and I can verify that the code to "uninstall" a security manager and install a new manager that uses the modified policy is in fact being executed. However, it looks like the new SecurityManager() is never actually being enforced. Or put another way, it looks like the security policy that takes effect is always the first one to be assigned in the JVM; subsequent attempts to set the System security manager to non-null values will re-enable the security manager, but will not alter its policy. Note, though, that attempts to UN-install the security manager (by calling System.setSecurityManager(null)) do work. But when we try try to call setSecurityManager() with a non-null value (namely, "new SecurityManager()"), the policy that it implements is always the first policy that was successfully applied for the JVM; if the "java.security.policy" property is pointing to some other (new) policy file, it is ignored. At least, this is what I've come to conclude based on my understanding of the security manager setup code. In order to get the binding test to work as part of a suite I can just uninstall the security manager altogether- that works fine. However, that seems a bit like cheating to me. And to be honest, I think that we should be able to change the security manager to use a new policy -or at least, that has been my impression based on the code comments. So my question, then, is this: does anyone know if I have to do something special to get the test to install a Security Manager that implements a different policy file than one that has previously been installed? I've spent a lot of time on this but have come up empty-handed. I'm attaching a patch, d1758_newSecMgr_doNotCommit.patch with the test setup changes that I've made. If there are any problems (glaring or otherwise) with my approach that might be leading to this inability to alter security policy, I would be grateful to anyone who could point them out... Note the patch includes all of the changes that I've made to add XMLBindingTest.java to a new XML JUnit suite. I'm hoping anyone who looks at it can just ignore the test-specific changes and focus on the security-related ones...
          Hide
          Daniel John Debrunner added a comment -

          Switching the security manager hasn't been tested, the first time I ran a test using the no security manager setup I had to fix a bug or two, so it's possible there are bugs in the code. Though maybe the permissions are fixed when the class is initally loaded, so changing the policy has no effect?

          The approach of enabling all code to read the DTD files is also "cheating". It removes the testing of the situation when one piece of the code in the stack doesn't have the permissions to read the DTD file, so it's just like running with no security manager.

          Show
          Daniel John Debrunner added a comment - Switching the security manager hasn't been tested, the first time I ran a test using the no security manager setup I had to fix a bug or two, so it's possible there are bugs in the code. Though maybe the permissions are fixed when the class is initally loaded, so changing the policy has no effect? The approach of enabling all code to read the DTD files is also "cheating". It removes the testing of the situation when one piece of the code in the stack doesn't have the permissions to read the DTD file, so it's just like running with no security manager.
          Hide
          A B added a comment -

          > The approach of enabling all code to read the DTD files is also "cheating".

          True. But my attempts to selectively give the JAXP parser permission to read the DTD file failed, as described in the comments for the change. Esp:

          + * Note that attempts to find the JAXP
          + * parser in the classpath as is done for the Derby jars
          + * in SecurityManagerSetup.determineClasspath() does not
          + * work--for whatever reason the "codeSource" for the
          + * JAXP parser classes is always null. Thus we cannot
          + * use a property like we do for the derby jars.

          I guess that should have been my first question, then: why is codeSource "null" and is there a way to get around it? If not, then allowing all code to read a single DTD file seems like "less" cheating than completely disabling the security manager altogether. But you're right, cheating is cheating...

          Show
          A B added a comment - > The approach of enabling all code to read the DTD files is also "cheating". True. But my attempts to selectively give the JAXP parser permission to read the DTD file failed, as described in the comments for the change. Esp: + * Note that attempts to find the JAXP + * parser in the classpath as is done for the Derby jars + * in SecurityManagerSetup.determineClasspath() does not + * work--for whatever reason the "codeSource" for the + * JAXP parser classes is always null. Thus we cannot + * use a property like we do for the derby jars. I guess that should have been my first question, then: why is codeSource "null" and is there a way to get around it? If not, then allowing all code to read a single DTD file seems like "less" cheating than completely disabling the security manager altogether. But you're right, cheating is cheating...
          Hide
          Daniel John Debrunner added a comment -

          I think having the XML test/suite running even without a security manager is a good step forward. As long as there is some issue in Jira (could be this one) that remains open until the security manager is enabled for the test then it's fine. "Progress, not perfection" in the open source world.

          You can use the SecurityManagerSetup.noSecurityManager() decorator method to disable the Securitymanager and it can be applied on a fixture by fixture basis, see DatabaseClassLoadingTest for an example, or it could be for a whole suite.

          Show
          Daniel John Debrunner added a comment - I think having the XML test/suite running even without a security manager is a good step forward. As long as there is some issue in Jira (could be this one) that remains open until the security manager is enabled for the test then it's fine. "Progress, not perfection" in the open source world. You can use the SecurityManagerSetup.noSecurityManager() decorator method to disable the Securitymanager and it can be applied on a fixture by fixture basis, see DatabaseClassLoadingTest for an example, or it could be for a whole suite.
          Hide
          Daniel John Debrunner added a comment -

          On the JAXP code source being null, I don't know the answer, but the real issue is how does an application grant permissions. Ideally the Derby documentation should cover this and again ideally Derby should not be requiring applications to grant permissions all the way up the stack. So the (maybe painful) process of discovering how to do this for the tests should make it easier for Derby user's to do the same. In this case because the parser can come from multiple places there may be multiple answers, e.g when the parser is bundled with the VM, when the parser is supplied by the application.

          Show
          Daniel John Debrunner added a comment - On the JAXP code source being null, I don't know the answer, but the real issue is how does an application grant permissions. Ideally the Derby documentation should cover this and again ideally Derby should not be requiring applications to grant permissions all the way up the stack. So the (maybe painful) process of discovering how to do this for the tests should make it easier for Derby user's to do the same. In this case because the parser can come from multiple places there may be multiple answers, e.g when the parser is bundled with the VM, when the parser is supplied by the application.
          Hide
          A B added a comment -

          Attaching a patch, d1758_newXMLSuite_v1.patch, that does the following:

          1. Updates XMLBindingTest to ignore the Windows line-ending character
          ("\r") when counting characters as part of serialization. See
          DERBY-2106 for more. This allows the test to run successfully
          regardless of platform.

          2. Updates XMLBindingTest to run with NO security manager for now,
          per Dan's suggestion. This works toward the "progress not
          perfection" goal of incremental development. Once the questions
          surrounding the security policy for JAXP have been answered the
          test can be updated to run with the security manager.

          3. Creates a new JUnit suite, suites/XMLSuite.java, to run all of the
          XML JUnit tests, and enables that suite to run as part of
          lang/_Suite.java, which in turn means it is executed as part
          suites.All.

          I ran lang/_Suite on Windows and Linux with ibm142 and ibm15 and there were no failures. I also ran the full "suites.All" on Windows with Sun jdk142 (using endorsed Xalan jars) with no failures. In all cases I used the "swingui" runner and verified manually that the XML tests ran and passed as expected. And finally, as a sanity check I ran the new XMLSuite on Windows with jdk142 (no endorsed jars) and jdk15 to verify that the XML tests do not run if the required classes are not present.

          Show
          A B added a comment - Attaching a patch, d1758_newXMLSuite_v1.patch, that does the following: 1. Updates XMLBindingTest to ignore the Windows line-ending character ("\r") when counting characters as part of serialization. See DERBY-2106 for more. This allows the test to run successfully regardless of platform. 2. Updates XMLBindingTest to run with NO security manager for now, per Dan's suggestion. This works toward the "progress not perfection" goal of incremental development. Once the questions surrounding the security policy for JAXP have been answered the test can be updated to run with the security manager. 3. Creates a new JUnit suite, suites/XMLSuite.java, to run all of the XML JUnit tests, and enables that suite to run as part of lang/_Suite.java, which in turn means it is executed as part suites.All. I ran lang/_Suite on Windows and Linux with ibm142 and ibm15 and there were no failures. I also ran the full "suites.All" on Windows with Sun jdk142 (using endorsed Xalan jars) with no failures. In all cases I used the "swingui" runner and verified manually that the XML tests ran and passed as expected. And finally, as a sanity check I ran the new XMLSuite on Windows with jdk142 (no endorsed jars) and jdk15 to verify that the XML tests do not run if the required classes are not present.
          Hide
          A B added a comment -

          Oh by the way, just to make it explicit, I committed d1758_newXBindTest_v1.patch to trunk on 11/17 with svn revision 476365.

          Show
          A B added a comment - Oh by the way, just to make it explicit, I committed d1758_newXBindTest_v1.patch to trunk on 11/17 with svn revision 476365.
          Hide
          Bryan Pendleton added a comment -

          Hi Army,

          I was successful applying d1758_newXMLSuite_v1.patch. I ran both XMLBindingTest and suites.XMLSuite with JDK 1.4.2 and Xalan 2.7 and the tests passed successfully. I also ran both XMLBindingTest and suites.XMLSuite with JDK 1.4.2 but without Xalan 2.7, and the tests quietly did nothing.

          In particular, when the tests passed, I did not see the #-of-characters diff that I had been seeing in my environment before. So I can confirm that that diff is gone for me.

          Show
          Bryan Pendleton added a comment - Hi Army, I was successful applying d1758_newXMLSuite_v1.patch. I ran both XMLBindingTest and suites.XMLSuite with JDK 1.4.2 and Xalan 2.7 and the tests passed successfully. I also ran both XMLBindingTest and suites.XMLSuite with JDK 1.4.2 but without Xalan 2.7, and the tests quietly did nothing. In particular, when the tests passed, I did not see the #-of-characters diff that I had been seeing in my environment before. So I can confirm that that diff is gone for me.
          Hide
          A B added a comment -

          Thank you for continuing to review the patches, Bryan, and for verifying that the new XMLBindingTest runs without error in your environment. I committed d1758_newXMLSuite_v1.patch with svn revision 478336.

          Show
          A B added a comment - Thank you for continuing to review the patches, Bryan, and for verifying that the new XMLBindingTest runs without error in your environment. I committed d1758_newXMLSuite_v1.patch with svn revision 478336.
          Hide
          A B added a comment -

          Attaching a patch to delete the old XML binding test and the old XML suite. Both have now been replaced with JUnit versions.

          The patch, d1758_remove_xBindAndSuite.patch, has been committed with svn 478348.

          Show
          A B added a comment - Attaching a patch to delete the old XML binding test and the old XML suite. Both have now been replaced with JUnit versions. The patch, d1758_remove_xBindAndSuite.patch, has been committed with svn 478348.
          Hide
          A B added a comment -

          Attaching a patch, d1758_secmgr_v1.patch, that enables the lang/XMLBindingTest to run under a security manager. Changes include all of the following:

          • Updates lang/XMLBindingTest.java so that it will run under the default testing security manager (i.e. removed the "noSecurityManager()" wrapper).
          • Adds a new property, derbyTesting.jaxpjar, to the default testing policy file. This property holds the location of the JAXP jar picked up from the classpath if that jar is external to the JVM. If the jar is either embedded within, or "endorsed" by, the JVM then this property is unused.

          The JAXP jar is then given permission to read the "extin" testing directory, which is the directory into which the DTD required by XMLBindingTest is copied (and thus JAXP has permission to read the DTD file).

          • Adds a new static utility method, "getJAXPParserLocation()", to the junit/XML.java file. This method instantiates a JAXP object and then uses the implementation-specific class name to try to find out where the JAXP jar is located.
          • Modifies derbyTesing/junit/build.xml so that junit/XML.java will only build with 1.4 JVMs and higher. This is required because junit/XML.java now references a JAXP class that is not defined in 1.3.
          • Updates the "getURL()" method of junit/SecurityManagerSetup.java to account for situations where a class "code source" is null. Also updates the "determineClasspath()" method of that class to set the derbyTesting.jaxpjar property as appropriate.
          • And finally, moves the build order of the derbyTesting/junit directory so that it is built before the derbyTesting/harness directory. This is required because the harness/jvm.java file references junit/SecurityManagerSetup.java, which in turn references junit/XML.java (with this patch). So when we try to compile the "harness" directory we will indirectly try to compile junit/XML.java using the harness/build.xml file--but that specific build file does not know that junit/XML.java is only supposed to be built with 1.4. By putting the junit target first we ensure that the junit/XML.java file is built with the junit/build.xml file, which knows not to use 1.3. At least, that's my understanding of the build process works. Someone please correct me if I'm wrong (and I very well could be...).

          I would appreciate reviews if anyone has the time (esp. anyone familiar with JUnit security manager testing and/or the build.xml files...)

          Show
          A B added a comment - Attaching a patch, d1758_secmgr_v1.patch, that enables the lang/XMLBindingTest to run under a security manager. Changes include all of the following: Updates lang/XMLBindingTest.java so that it will run under the default testing security manager (i.e. removed the "noSecurityManager()" wrapper). Adds a new property, derbyTesting.jaxpjar, to the default testing policy file. This property holds the location of the JAXP jar picked up from the classpath if that jar is external to the JVM. If the jar is either embedded within, or "endorsed" by, the JVM then this property is unused. The JAXP jar is then given permission to read the "extin" testing directory, which is the directory into which the DTD required by XMLBindingTest is copied (and thus JAXP has permission to read the DTD file). Adds a new static utility method, "getJAXPParserLocation()", to the junit/XML.java file. This method instantiates a JAXP object and then uses the implementation-specific class name to try to find out where the JAXP jar is located. Modifies derbyTesing/junit/build.xml so that junit/XML.java will only build with 1.4 JVMs and higher. This is required because junit/XML.java now references a JAXP class that is not defined in 1.3. Updates the "getURL()" method of junit/SecurityManagerSetup.java to account for situations where a class "code source" is null. Also updates the "determineClasspath()" method of that class to set the derbyTesting.jaxpjar property as appropriate. And finally, moves the build order of the derbyTesting/junit directory so that it is built before the derbyTesting/harness directory. This is required because the harness/jvm.java file references junit/SecurityManagerSetup.java, which in turn references junit/XML.java (with this patch). So when we try to compile the "harness" directory we will indirectly try to compile junit/XML.java using the harness/build.xml file--but that specific build file does not know that junit/XML.java is only supposed to be built with 1.4. By putting the junit target first we ensure that the junit/XML.java file is built with the junit/build.xml file, which knows not to use 1.3. At least, that's my understanding of the build process works. Someone please correct me if I'm wrong (and I very well could be...). I would appreciate reviews if anyone has the time (esp. anyone familiar with JUnit security manager testing and/or the build.xml files...)
          Hide
          Andrew McIntyre added a comment -
          • Modifies derbyTesing/junit/build.xml so that junit/XML.java will only build with 1.4 JVMs and higher. This is required because junit/XML.java now references a JAXP class that is not defined in 1.3.

          I took a look at the build file changes in this patch. Could you move the addition of $

          {xml-apis}

          into a separate pathelement? A minor nitpick, i know, but I just ran into some problems trying to automate some buildfile munging and got snagged when a classpath element I was trying to remove was in the same pathelement as the compile.classpath variable.

          Show
          Andrew McIntyre added a comment - Modifies derbyTesing/junit/build.xml so that junit/XML.java will only build with 1.4 JVMs and higher. This is required because junit/XML.java now references a JAXP class that is not defined in 1.3. I took a look at the build file changes in this patch. Could you move the addition of $ {xml-apis} into a separate pathelement? A minor nitpick, i know, but I just ran into some problems trying to automate some buildfile munging and got snagged when a classpath element I was trying to remove was in the same pathelement as the compile.classpath variable.
          Hide
          Bryan Pendleton added a comment -

          I applied and built the change, and ran XLMBindingTest and XMLSuite under Sun 1.4.2
          both with and without Xalan 2.7 as an endorsed extension. Tests ran fine. No problems to report.

          Show
          Bryan Pendleton added a comment - I applied and built the change, and ran XLMBindingTest and XMLSuite under Sun 1.4.2 both with and without Xalan 2.7 as an endorsed extension. Tests ran fine. No problems to report.
          Hide
          A B added a comment -

          Thank you for the reviews Bryan and Andrew. I made the build change that Andrew suggested and committed the patch with svn 482433. I'll wait another day or two to see if there are any follow-up suggestions/corrections before resolving and closing this issue.

          I also updated the summary/description of this issue to make it agree more closely with the final changes.

          Show
          A B added a comment - Thank you for the reviews Bryan and Andrew. I made the build change that Andrew suggested and committed the patch with svn 482433. I'll wait another day or two to see if there are any follow-up suggestions/corrections before resolving and closing this issue. I also updated the summary/description of this issue to make it agree more closely with the final changes.
          Hide
          A B added a comment -

          Final patch checked in on 12/04. One failure resulted from the changes but that was filed/fixed as DERBY-2153. So resolving/closing this issue.

          Show
          A B added a comment - Final patch checked in on 12/04. One failure resulted from the changes but that was filed/fixed as DERBY-2153 . So resolving/closing this issue.

            People

            • Assignee:
              A B
              Reporter:
              A B
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development