Derby
  1. Derby
  2. DERBY-590

How to integrate Derby with Lucene API?

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SQL
    • Urgency:
      Low

      Description

      In order to use derby with lucene API what should be the steps to be taken?

      1. netbeans2.diff
        2 kB
        Knut Anders Hatlen
      2. netbeans.diff
        3 kB
        Knut Anders Hatlen
      3. multifield-with-custom-tokenizers.diff
        17 kB
        Knut Anders Hatlen
      4. multifield.diff
        18 kB
        Knut Anders Hatlen
      5. LucenePlugin.html
        22 kB
        Rick Hillegas
      6. LucenePlugin.html
        22 kB
        Rick Hillegas
      7. LucenePlugin.html
        23 kB
        Rick Hillegas
      8. LucenePlugin.html
        23 kB
        Rick Hillegas
      9. lucene_demo.diff
        23 kB
        Andrew McIntyre
      10. lucene_demo_2.diff
        41 kB
        Andrew McIntyre
      11. exceptions.diff
        27 kB
        Knut Anders Hatlen
      12. derby-590-32-aa-vetFieldNames.diff
        8 kB
        Rick Hillegas
      13. derby-590-31-aa-multiField.diff
        66 kB
        Rick Hillegas
      14. derby-590-30-aa-nullableScoreCeiling.diff
        12 kB
        Rick Hillegas
      15. derby-590-29-aa-useLucene_4.7.1.diff
        3 kB
        Rick Hillegas
      16. derby-590-28-renameLuceneJars.diff
        1 kB
        Rick Hillegas
      17. derby-590-27-aa-publicAPILuceneUtils.diff
        0.8 kB
        Rick Hillegas
      18. derby-590-26-ad-backupRestoreEncryption.diff
        40 kB
        Rick Hillegas
      19. derby-590-26-ac-backupRestore.diff
        32 kB
        Rick Hillegas
      20. derby-590-24-ad-luceneDirectory.diff
        63 kB
        Rick Hillegas
      21. derby-590-23-aa-correctTestLocale.diff
        4 kB
        Rick Hillegas
      22. derby-590-22-aa-cleanupPrivacy.diff
        11 kB
        Rick Hillegas
      23. derby-590-21-aa-noTimeTravel.diff
        5 kB
        Rick Hillegas
      24. derby-590-20-aa-customQueryParser.diff
        22 kB
        Rick Hillegas
      25. derby-590-19-aa-cleanupAPI2.diff
        10 kB
        Rick Hillegas
      26. derby-590-18-aa-cleanupAPI.diff
        15 kB
        Rick Hillegas
      27. derby-590-17-aa-closeInputStreamOnPropertiesFile.diff
        2 kB
        Rick Hillegas
      28. derby-590-16-aa-adjustUpgradeTest.diff
        3 kB
        Rick Hillegas
      29. derby-590-15-aa-requireHardUpgrade.diff
        4 kB
        Rick Hillegas
      30. derby-590-14-aa-coarseGrainedAuthorization.diff
        17 kB
        Rick Hillegas
      31. derby-590-13-aa-indexViews.diff
        17 kB
        Rick Hillegas
      32. derby-590-12-aa-newJar.diff
        7 kB
        Rick Hillegas
      33. derby-590-11-aa-moveCode.diff
        192 kB
        Rick Hillegas
      34. derby-590-10-aa-fixLocaleTest.diff
        0.8 kB
        Rick Hillegas
      35. derby-590-09-aa-localeSensitiveAnalysis.diff
        53 kB
        Rick Hillegas
      36. derby-590-08-aa-omitLuceneFlag.diff
        4 kB
        Rick Hillegas
      37. derby-590-07-aa-accessClassInPackage.sun.misc.diff
        1 kB
        Rick Hillegas
      38. derby-590-06-aa-suppressAccessChecks.diff
        1.0 kB
        Rick Hillegas
      39. derby-590-05-aa-accessDeclaredMembers.diff
        2 kB
        Rick Hillegas
      40. derby-590-04-aa-removeIDFromListIndexes.diff
        6 kB
        Rick Hillegas
      41. derby-590-03-aa-removeTestingDiagnostic.diff
        0.5 kB
        Rick Hillegas
      42. derby-590-02-aa-cleanupFindbugsErrors.diff
        2 kB
        Rick Hillegas
      43. derby-590-01-am-publicAccessToLuceneRoutines.diff
        174 kB
        Rick Hillegas
      44. derby-590-01-ah-publicAccessToLuceneRoutines.diff
        129 kB
        Rick Hillegas
      45. derby-590-01-ag-publicAccessToLuceneRoutines.diff
        81 kB
        Rick Hillegas

        Issue Links

          Activity

          Hide
          Kim Haase added a comment -

          Not a doc issue. Related doc issue is DERBY-6564.

          Show
          Kim Haase added a comment - Not a doc issue. Related doc issue is DERBY-6564 .
          Hide
          Knut Anders Hatlen added a comment -

          Thanks for making these changes, Rick. The new signatures seem to provide the flexibility needed. At least the new API was sufficiently flexible for a pointless toy application I wrote to test support for multiple fields...

          Show
          Knut Anders Hatlen added a comment - Thanks for making these changes, Rick. The new signatures seem to provide the flexibility needed. At least the new API was sufficiently flexible for a pointless toy application I wrote to test support for multiple fields...
          Hide
          Rick Hillegas added a comment -

          Attaching a new version of the functional spec. This version incorporates the changes which Knut and I discussed:

          o The signatures of createIndex() and updateIndex() have changed. The analyzerMaker argument has been replaced by an indexDescriptorMaker argument.
          
          o Corresponding changes have been made to the examples for createIndex() and updateIndex().
          
          o The queryParserMaker argument has been removed from the query table function.
          
          o Corresponding changes have been made to the query examples.
          
          o The analyzerMaker column has been replaced with an indexDescriptorMaker column in the result returned by the listIndexes() table function.
          
          o The sample javadoc for LuceneUtils has changed:
          
            - Added a new method: defaultIndexDescriptor()
          
            - Changed the argument signature of defaultQueryParser()
          
            - Changed the exception signatures of defaultAnalyzer() and getAnalyzerForLocale()
          
          o A new interface has been added to the public API section: IndexDescriptor
          
          Show
          Rick Hillegas added a comment - Attaching a new version of the functional spec. This version incorporates the changes which Knut and I discussed: o The signatures of createIndex() and updateIndex() have changed. The analyzerMaker argument has been replaced by an indexDescriptorMaker argument. o Corresponding changes have been made to the examples for createIndex() and updateIndex(). o The queryParserMaker argument has been removed from the query table function. o Corresponding changes have been made to the query examples. o The analyzerMaker column has been replaced with an indexDescriptorMaker column in the result returned by the listIndexes() table function. o The sample javadoc for LuceneUtils has changed: - Added a new method: defaultIndexDescriptor() - Changed the argument signature of defaultQueryParser() - Changed the exception signatures of defaultAnalyzer() and getAnalyzerForLocale() o A new interface has been added to the public API section: IndexDescriptor
          Hide
          ASF subversion and git services added a comment -

          Commit 1606719 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1606719 ]

          DERBY-590: Add checks for legal field names for Lucene indexes; commit derby-590-32-aa-vetFieldNames.diff.

          Show
          ASF subversion and git services added a comment - Commit 1606719 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1606719 ] DERBY-590 : Add checks for legal field names for Lucene indexes; commit derby-590-32-aa-vetFieldNames.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-32-aa-vetFieldNames.diff. This patch adds checks for legal field names. The Lucene tests pass cleanly for me on this patch.

          This patch enforces the following rules:

          1) Fields and keys may not have the same names.

          2) A field name may not be null.

          3) Duplicate field names are not allowed.

          Touches the following files:

          --------------

          M java/engine/org/apache/derby/loc/messages.xml
          M java/shared/org/apache/derby/shared/common/reference/SQLState.java

          --------------

          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java

          Enforcement code.

          --------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java

          Tests.

          Show
          Rick Hillegas added a comment - Attaching derby-590-32-aa-vetFieldNames.diff. This patch adds checks for legal field names. The Lucene tests pass cleanly for me on this patch. This patch enforces the following rules: 1) Fields and keys may not have the same names. 2) A field name may not be null. 3) Duplicate field names are not allowed. Touches the following files: -------------- M java/engine/org/apache/derby/loc/messages.xml M java/shared/org/apache/derby/shared/common/reference/SQLState.java -------------- M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java Enforcement code. -------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java Tests.
          Hide
          ASF subversion and git services added a comment -

          Commit 1606161 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1606161 ]

          DERBY-590: Add multi-field support to the Lucene plugin; commit derby-590-31-aa-multiField.diff.

          Show
          ASF subversion and git services added a comment - Commit 1606161 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1606161 ] DERBY-590 : Add multi-field support to the Lucene plugin; commit derby-590-31-aa-multiField.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-xx-publicAPILuceneUtils.diff. This patch changes the api of the Lucene plugin as Knut and I discussed. The Lucene tests run cleanly with this patch.

          This patch makes the following changes:

          1) Adds a new interface to the public api: org.apache.derby.optional.api.LuceneIndexDescriptor. This interface allows the application developer to describe the following:

          a) The names of fields which can be specified in Lucene queries.

          b) The Analyzer which should be used to build the index.

          c) The QueryParser which should be used to parse Lucene queries against the index.

          2) Changes the signatures of createIndex() and updateIndex(). The routines now take an indexDescriptorMaker rather than an analyzerMaker.

          3) Changes the signature of the index-specific table function which is used to run queries against the index. The queryParserMaker argument has been removed. Internally, the query uses the QueryParser returned by the LuceneIndexDescriptor which was bound to the index at createIndex() time.

          4) Changes the signature of the table returned by listIndexes(). The table no longer has an analyzerMaker column. Instead, it has an indexDescriptorMaker column.

          I expect that I will file a follow-on patch to address some more issues which I want to explore. I think that we need to prevent a LuceneIndexDescriptor from specifying field names which conflict with the names of the key and text columns.

          After that, I plan to update the functional spec and file a doc issue to address these changes.

          Touches the following files:

          --------------------

          A java/optional/org/apache/derby/optional/api/LuceneIndexDescriptor.java
          M java/optional/org/apache/derby/optional/api/LuceneUtils.java
          M tools/javadoc/publishedapi.ant

          Support for the new interface in the public api.

          --------------------

          M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          M java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java

          Signature changes. Support for multi-field queries.

          --------------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/myLuceneClasses.jar
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneBackupTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneInMemoryTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneJarLoadingTest.java

          Adjusted tests to handle the new api. Added Knut's multi-field test case.

          Show
          Rick Hillegas added a comment - Attaching derby-590-xx-publicAPILuceneUtils.diff. This patch changes the api of the Lucene plugin as Knut and I discussed. The Lucene tests run cleanly with this patch. This patch makes the following changes: 1) Adds a new interface to the public api: org.apache.derby.optional.api.LuceneIndexDescriptor. This interface allows the application developer to describe the following: a) The names of fields which can be specified in Lucene queries. b) The Analyzer which should be used to build the index. c) The QueryParser which should be used to parse Lucene queries against the index. 2) Changes the signatures of createIndex() and updateIndex(). The routines now take an indexDescriptorMaker rather than an analyzerMaker. 3) Changes the signature of the index-specific table function which is used to run queries against the index. The queryParserMaker argument has been removed. Internally, the query uses the QueryParser returned by the LuceneIndexDescriptor which was bound to the index at createIndex() time. 4) Changes the signature of the table returned by listIndexes(). The table no longer has an analyzerMaker column. Instead, it has an indexDescriptorMaker column. I expect that I will file a follow-on patch to address some more issues which I want to explore. I think that we need to prevent a LuceneIndexDescriptor from specifying field names which conflict with the names of the key and text columns. After that, I plan to update the functional spec and file a doc issue to address these changes. Touches the following files: -------------------- A java/optional/org/apache/derby/optional/api/LuceneIndexDescriptor.java M java/optional/org/apache/derby/optional/api/LuceneUtils.java M tools/javadoc/publishedapi.ant Support for the new interface in the public api. -------------------- M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java M java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java Signature changes. Support for multi-field queries. -------------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/myLuceneClasses.jar M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneBackupTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneInMemoryTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneJarLoadingTest.java Adjusted tests to handle the new api. Added Knut's multi-field test case.
          Hide
          Knut Anders Hatlen added a comment -

          Hi Rick,

          I basically agree with you.

          1) It's a bit awkward, yes. One way to avoid it, might be to allow users to specify a custom TokenStream class for each field. I tried that approach in the patch multifield-with-custom-tokenizers.diff. With that patch, you specify the fields and the tokenizers in derby.tests.lucene.fields property like this: field1:nameOfTokenizerClass1,field2:nameOfTokenizerClass2,... If a tokenizer has been specified for the field, LuceneSupport.createOrRecreateIndex() uses the TextField constructor that takes a TokenStream instead of the constructor that takes a String. With that approach, neither a custom analyzer nor a custom query parser is needed in this particular use case.

          2) Even without adding support for multiple fields, I think it sounds like an improvement to allow specifying both analyzer and query parser at index-creation time. Sounds more convenient than repeating the information about the query parser on each call to the query table function. I don't have enough experience with Lucene to say if the query parser parameter should be removed from the table function completely, or if we should keep the ability to override the query parser for individual queries. My guess is that it's OK to remove the parameter and only allow one query parser per index.

          Show
          Knut Anders Hatlen added a comment - Hi Rick, I basically agree with you. 1) It's a bit awkward, yes. One way to avoid it, might be to allow users to specify a custom TokenStream class for each field. I tried that approach in the patch multifield-with-custom-tokenizers.diff . With that patch, you specify the fields and the tokenizers in derby.tests.lucene.fields property like this: field1:nameOfTokenizerClass1,field2:nameOfTokenizerClass2,... If a tokenizer has been specified for the field, LuceneSupport.createOrRecreateIndex() uses the TextField constructor that takes a TokenStream instead of the constructor that takes a String. With that approach, neither a custom analyzer nor a custom query parser is needed in this particular use case. 2) Even without adding support for multiple fields, I think it sounds like an improvement to allow specifying both analyzer and query parser at index-creation time. Sounds more convenient than repeating the information about the query parser on each call to the query table function. I don't have enough experience with Lucene to say if the query parser parameter should be removed from the table function completely, or if we should keep the ability to override the query parser for individual queries. My guess is that it's OK to remove the parameter and only allow one query parser per index.
          Hide
          Rick Hillegas added a comment - - edited

          Hi Knut,

          I think this is a promising approach. I am a little worried about a couple things:

          1) The awkwardness of using one Analyzer to tokenize the document and another Analyzer to tokenize the query. It's tempting to blame Lucene for mis-factoring this. Maybe this is unavoidable since Lucene lets you write your own custom query language. But disturbingly, this may indicate that I don't know what I'm talking about and have mis-factored Derby.

          2) The increasing wordiness of the Derby api. Maybe we need to make the user declare fields, Analyzer, and QueryParser all at create/update index time. E.g., introduce an interface like this:

          public interface IndexDescriptor
          {
              public String[] getFields();
              public Analyzer getAnalyzer();
              public QueryParser getQueryParser();
          }
          

          There would be a default IndexDescriptor for the main use case.

          Then we could change the signature of createIndex() to…

          LUCENESUPPORT.CREATEINDEX
          (
              SCHEMANAME VARCHAR( 128 ),
              TABLENAME VARCHAR( 128 ),
              TEXTCOLUMN VARCHAR( 128 ),
              INDEXDESCRIPTORMAKER VARCHAR( 32672 ),
              KEYCOLUMNS VARCHAR( 32672 ) ...
          )
          

          ...and simplify the signature of the query table function to...

          $SCHEMANAME.$TABLENAME__TEXTCOL
          (
              QUERY VARCHAR( 32672 ),
              WINDOWSIZE INT,
              SCORECEILING REAL
          )
          RETURNS TABLE
          (
              $keyColumn1 $keyColumn1datatype,
              ...
              $keyColumnN $keyColumnNdatatype,
              DOCUMENTID INT,
              SCORE REAL
          )
          
          

          If nothing else, this helps future-proof us against other changes to the api which real users will suggest as they experiment with the plugin. It also means that the user has to declare only one static method to materialize the index context.

          What do you think?

          Show
          Rick Hillegas added a comment - - edited Hi Knut, I think this is a promising approach. I am a little worried about a couple things: 1) The awkwardness of using one Analyzer to tokenize the document and another Analyzer to tokenize the query. It's tempting to blame Lucene for mis-factoring this. Maybe this is unavoidable since Lucene lets you write your own custom query language. But disturbingly, this may indicate that I don't know what I'm talking about and have mis-factored Derby. 2) The increasing wordiness of the Derby api. Maybe we need to make the user declare fields, Analyzer, and QueryParser all at create/update index time. E.g., introduce an interface like this: public interface IndexDescriptor { public String[] getFields(); public Analyzer getAnalyzer(); public QueryParser getQueryParser(); } There would be a default IndexDescriptor for the main use case. Then we could change the signature of createIndex() to… LUCENESUPPORT.CREATEINDEX ( SCHEMANAME VARCHAR( 128 ), TABLENAME VARCHAR( 128 ), TEXTCOLUMN VARCHAR( 128 ), INDEXDESCRIPTORMAKER VARCHAR( 32672 ), KEYCOLUMNS VARCHAR( 32672 ) ... ) ...and simplify the signature of the query table function to... $SCHEMANAME.$TABLENAME__TEXTCOL ( QUERY VARCHAR( 32672 ), WINDOWSIZE INT, SCORECEILING REAL ) RETURNS TABLE ( $keyColumn1 $keyColumn1datatype, ... $keyColumnN $keyColumnNdatatype, DOCUMENTID INT, SCORE REAL ) If nothing else, this helps future-proof us against other changes to the api which real users will suggest as they experiment with the plugin. It also means that the user has to declare only one static method to materialize the index context. What do you think?
          Hide
          Knut Anders Hatlen added a comment -

          Thanks, Rick. Those were the exact changes that were needed.

          The attached patch multifield.diff shows an example of how it could be used.

          I made two small adjustments:

          1) Instead of hard-coding the field names, I made LuceneSupport read them dynamically from a database property (derby.tests.lucene.fields), so that I could verify that the original Lucene tests still pass. (They do still pass, by the way.) Also the field names are stored in the Lucene index property file, so that LuceneQueryVTI can find them too. This is of course just a temporary hack until we figure out the correct API.

          2) I made LuceneUtils.defaultQueryParser() always return a MultiFieldQueryParser, since MultiFieldQueryParser seems to behave just like QueryParser in the degenerate case with a single field.

          Since I didn't feel like writing a Java source file parser, I changed my example use case to search in XML files, so that I could use the XML parser that is in the JRE. I added a test case to LuceneSupportTest to verify that it could be used for that.

          The test case creates an index with two fields: tags and text. The tags field contains only the XML tags, whereas the text field contains only the text elements of the XML file. This way, you can use the index to search for data and metadata separately in the XML documents stored in your table.

          Now, while writing the test case, I found that you will most likely want to use a custom query parser when you use it this way. The reason is that the default query parser uses the same analyzer as the index writer used to extract tokens from the search terms. That means, if you like in this case use a custom analyzer that parser XML documents, the query parser will also expect the terms in the query to be XML documents. So you'll end up with rather silly-looking queries.

          For example, to search for documents that contain the text "abc", you cannot make the query text:"abc", but have to wrap it in dummy XML tags to make it parsable text:"<dummy>abc</dummy>".

          The custom query parser doesn't need to be very complex, though. The test case in the patch shows one example in the method createXMLQueryParser(). That method simply creates a MultiFieldQueryParser with a plain StandardAnalyzer. With that parser, you can write queries like:

          • text:abc to search for "abc" in the text elements of the XML
          • tags:abc to search for XML tags called "abc"
          • abc to search for "abc" in both text elements and tags

          What do you think? Does it sound like a useful addition?

          Show
          Knut Anders Hatlen added a comment - Thanks, Rick. Those were the exact changes that were needed. The attached patch multifield.diff shows an example of how it could be used. I made two small adjustments: 1) Instead of hard-coding the field names, I made LuceneSupport read them dynamically from a database property (derby.tests.lucene.fields), so that I could verify that the original Lucene tests still pass. (They do still pass, by the way.) Also the field names are stored in the Lucene index property file, so that LuceneQueryVTI can find them too. This is of course just a temporary hack until we figure out the correct API. 2) I made LuceneUtils.defaultQueryParser() always return a MultiFieldQueryParser, since MultiFieldQueryParser seems to behave just like QueryParser in the degenerate case with a single field. Since I didn't feel like writing a Java source file parser, I changed my example use case to search in XML files, so that I could use the XML parser that is in the JRE. I added a test case to LuceneSupportTest to verify that it could be used for that. The test case creates an index with two fields: tags and text. The tags field contains only the XML tags, whereas the text field contains only the text elements of the XML file. This way, you can use the index to search for data and metadata separately in the XML documents stored in your table. Now, while writing the test case, I found that you will most likely want to use a custom query parser when you use it this way. The reason is that the default query parser uses the same analyzer as the index writer used to extract tokens from the search terms. That means, if you like in this case use a custom analyzer that parser XML documents, the query parser will also expect the terms in the query to be XML documents. So you'll end up with rather silly-looking queries. For example, to search for documents that contain the text "abc", you cannot make the query text:"abc" , but have to wrap it in dummy XML tags to make it parsable text:"<dummy>abc</dummy>" . The custom query parser doesn't need to be very complex, though. The test case in the patch shows one example in the method createXMLQueryParser() . That method simply creates a MultiFieldQueryParser with a plain StandardAnalyzer. With that parser, you can write queries like: text:abc to search for "abc" in the text elements of the XML tags:abc to search for XML tags called "abc" abc to search for "abc" in both text elements and tags What do you think? Does it sound like a useful addition?
          Hide
          Rick Hillegas added a comment -

          Hi Knut,

          I think I understand what you want now. You could try prototyping this to see if it works. Try this:

          Where HARD_CODED_FIELD_NAMES is the array of field names you want, e.g.,

          { "comment", "method" }

          ):

          1) In the loop at the end of LuceneSupport.createOrRecreateIndex(), replace...

                  if ( textcolValue != null )
                  {
                      doc.add(new TextField( LuceneQueryVTI.TEXT_FIELD_NAME, textcolValue, Store.NO));
                  }
          

          ...with...

                  if ( textcolValue != null )
                  {
                      for ( String fieldName : HARD_CODED_FIELD_NAMES )
                      {
                          doc.add(new TextField( fieldName, textcolValue, Store.NO));
                      }
                  }
          

          2) In LuceneQueryVTI, replace the getQueryParser() method with...

          	private static QueryParser getQueryParser
                  (
                   final String queryParserMaker,
                   final Version version,
                   final String fieldName,
                   final Analyzer analyzer
                   )
              {
                  return new MultiFieldQueryParser( version, HARD_CODED_FIELD_NAMES, analyzer );
              }
          
          Show
          Rick Hillegas added a comment - Hi Knut, I think I understand what you want now. You could try prototyping this to see if it works. Try this: Where HARD_CODED_FIELD_NAMES is the array of field names you want, e.g., { "comment", "method" } ): 1) In the loop at the end of LuceneSupport.createOrRecreateIndex(), replace... if ( textcolValue != null ) { doc.add(new TextField( LuceneQueryVTI.TEXT_FIELD_NAME, textcolValue, Store.NO)); } ...with... if ( textcolValue != null ) { for ( String fieldName : HARD_CODED_FIELD_NAMES ) { doc.add(new TextField( fieldName, textcolValue, Store.NO)); } } 2) In LuceneQueryVTI, replace the getQueryParser() method with... private static QueryParser getQueryParser ( final String queryParserMaker, final Version version, final String fieldName, final Analyzer analyzer ) { return new MultiFieldQueryParser( version, HARD_CODED_FIELD_NAMES, analyzer ); }
          Hide
          Knut Anders Hatlen added a comment -

          I suppose you could simulate the functionality that way. You'd probably need a custom query parser as well, in that case, in order to make the query language understand that "method:compute" is a single token. In the default Lucene query parser, that would be interpreted as a search for the token "compute" in the field "method".

          By the way, when I said "multiple indexes" and "multiple analyzers" above, I think I meant what in Lucene speak should have been "multiple fields". I think it's still called a single index in Lucene speak, even if you index separately on multiple fields/keys.

          Currently, when the luceneSupport tool creates an index, it makes every string value a Document with a single field called "luceneTextField".

          LuceneSupport.java#createOrRecreateIndex
                          String  textcolValue = rs.getString( keyCount + 1 );
                          if ( textcolValue != null )
                          {
                              doc.add(new TextField( LuceneQueryVTI.TEXT_FIELD_NAME, textcolValue, Store.NO));
                          }
                          addDocument( iw, doc );
          

          The flexibility I was looking for, was the ability to have more fields than the single, hard-coded one. For example, by having an extra argument to CREATEINDEX (and UPDATEINDEX) which is a comma-separated list of field names (with a reasonable default when NULL), and make the above code add each of the fields.

          In my hypothetical Java code in a CLOB example, that would mean something like this for creating the index:

          CALL LUCENESUPPORT.CREATEINDEX('app', 'sourcefiles', 'sourcetext', 'MyAnalyzer.create', 'comment,method', 'pk')
          

          The custom analyzer would be something like this:

          public class MyAnalyzer extends Analyzer {
          
              public static Analyzer create() {
                  return new MyAnalyzer();
              }
          
              @Override
              protected TokenStreamComponents createComponents(String field, Reader r) {
                  switch (field) {
                      case "comment":
                          return new TokenStreamComponents(createCommentTokenizer(r));
                      case "method":
                          return new TokenStreamComponents(createMethodTokenizer(r));
                      default:
                          throw new AssertionError("unknown field name: " + field);
                  }
              }
          
              private static Tokenizer createCommentTokenizer(Reader r) {
                  // TODO: Create a tokenizer that extracts tokens only from
                  // code comments.
                  // ....
              }
          
              private static Tokenizer createMethodTokenizer(Reader r) {
                  // TODO: Create a tokenizer that only returns method names.
                  // ....
              }
          
          }
          

          Might not add any functionality that you couldn't work around somehow with the current implementation. But I think that the extra flexibility would allow the application to push more of the full-text search logic down to Lucene, where it belongs. At least you'd avoid the need for a custom query parser and creation of synthetic tokens.

          Show
          Knut Anders Hatlen added a comment - I suppose you could simulate the functionality that way. You'd probably need a custom query parser as well, in that case, in order to make the query language understand that "method:compute" is a single token. In the default Lucene query parser, that would be interpreted as a search for the token "compute" in the field "method". By the way, when I said "multiple indexes" and "multiple analyzers" above, I think I meant what in Lucene speak should have been "multiple fields". I think it's still called a single index in Lucene speak, even if you index separately on multiple fields/keys. Currently, when the luceneSupport tool creates an index, it makes every string value a Document with a single field called "luceneTextField". LuceneSupport.java#createOrRecreateIndex String textcolValue = rs.getString( keyCount + 1 ); if ( textcolValue != null ) { doc.add( new TextField( LuceneQueryVTI.TEXT_FIELD_NAME, textcolValue, Store.NO)); } addDocument( iw, doc ); The flexibility I was looking for, was the ability to have more fields than the single, hard-coded one. For example, by having an extra argument to CREATEINDEX (and UPDATEINDEX) which is a comma-separated list of field names (with a reasonable default when NULL), and make the above code add each of the fields. In my hypothetical Java code in a CLOB example, that would mean something like this for creating the index: CALL LUCENESUPPORT.CREATEINDEX('app', 'sourcefiles', 'sourcetext', 'MyAnalyzer.create', 'comment,method', 'pk') The custom analyzer would be something like this: public class MyAnalyzer extends Analyzer { public static Analyzer create() { return new MyAnalyzer(); } @Override protected TokenStreamComponents createComponents( String field, Reader r) { switch (field) { case "comment" : return new TokenStreamComponents(createCommentTokenizer(r)); case "method" : return new TokenStreamComponents(createMethodTokenizer(r)); default : throw new AssertionError( "unknown field name: " + field); } } private static Tokenizer createCommentTokenizer(Reader r) { // TODO: Create a tokenizer that extracts tokens only from // code comments. // .... } private static Tokenizer createMethodTokenizer(Reader r) { // TODO: Create a tokenizer that only returns method names. // .... } } Might not add any functionality that you couldn't work around somehow with the current implementation. But I think that the extra flexibility would allow the application to push more of the full-text search logic down to Lucene, where it belongs. At least you'd avoid the need for a custom query parser and creation of synthetic tokens.
          Hide
          Rick Hillegas added a comment - - edited

          Hi Knut,

          I don't think I understand the problem case you're working on, but it sounds interesting. Could the problem be tackled by using one index and by writing a custom Analyzer which produces index terms like "comment:TODO" and "method:compute"? Thanks.

          Show
          Rick Hillegas added a comment - - edited Hi Knut, I don't think I understand the problem case you're working on, but it sounds interesting. Could the problem be tackled by using one index and by writing a custom Analyzer which produces index terms like "comment:TODO" and "method:compute"? Thanks.
          Hide
          Knut Anders Hatlen added a comment -

          Thanks, Rick.

          I don't think the lack of cascade on DROP TABLE is a big deal. It would be nice to have, but the users will understand what to do in any case.

          Using a VIEW for allowing multiple analyzers on a single column sounds like a useful workaround in many cases. That probably means you'll have to do the combining and scoring of the results in SQL instead of in Lucene, though. That is, in the Java source code example mentioned above, you cannot give a query string such as "comment:TODO method:compute" to find source files that have a method called "compute" and a comment that says "TODO". Instead, you'll need two separate queries, and some logic for combining the results. I'm assuming Lucene knows better how to do that.

          Show
          Knut Anders Hatlen added a comment - Thanks, Rick. I don't think the lack of cascade on DROP TABLE is a big deal. It would be nice to have, but the users will understand what to do in any case. Using a VIEW for allowing multiple analyzers on a single column sounds like a useful workaround in many cases. That probably means you'll have to do the combining and scoring of the results in SQL instead of in Lucene, though. That is, in the Java source code example mentioned above, you cannot give a query string such as "comment:TODO method:compute" to find source files that have a method called "compute" and a comment that says "TODO". Instead, you'll need two separate queries, and some logic for combining the results. I'm assuming Lucene knows better how to do that.
          Hide
          Rick Hillegas added a comment -

          Hi Knut,

          Thanks for kicking the tires! Here are some responses to your questions:

          KAH> Is there a way to create multiple Lucene indexes on one column?

          You should be able to do this by defining a view on the base table and then creating a Lucene index on the view.

          KAH> Would it be difficult to make the luceneSupport tool capable of using custom analyzers in jar files installed with SQLJ.INSTALL_JAR? I get a ClassNotFoundException if I try that now.

          That should be easy to fix. Sounds like the code needs to use the database class loader when resolving analyzers and query parsers. I have logged DERBY-6600 to track this. Thanks for pointing this out.

          KAH> With the current architecture, is it possible to make DROP TABLE also drop dependent Lucene indexes? Or would the luceneSupport tool have to be moved into the engine in order to have access to the dependency manager?

          This is a defect of a loosely coupled approach. Altering a Lucene-indexed table won't affect the Lucene indexes. The tool can probably get its hands on the dependency manager, though. Would need some investigation.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Knut, Thanks for kicking the tires! Here are some responses to your questions: KAH> Is there a way to create multiple Lucene indexes on one column? You should be able to do this by defining a view on the base table and then creating a Lucene index on the view. KAH> Would it be difficult to make the luceneSupport tool capable of using custom analyzers in jar files installed with SQLJ.INSTALL_JAR? I get a ClassNotFoundException if I try that now. That should be easy to fix. Sounds like the code needs to use the database class loader when resolving analyzers and query parsers. I have logged DERBY-6600 to track this. Thanks for pointing this out. KAH> With the current architecture, is it possible to make DROP TABLE also drop dependent Lucene indexes? Or would the luceneSupport tool have to be moved into the engine in order to have access to the dependency manager? This is a defect of a loosely coupled approach. Altering a Lucene-indexed table won't affect the Lucene indexes. The tool can probably get its hands on the dependency manager, though. Would need some investigation. Thanks, -Rick
          Hide
          Knut Anders Hatlen added a comment -

          With the current architecture, is it possible to make DROP TABLE also drop dependent Lucene indexes? Or would the luceneSupport tool have to be moved into the engine in order to have access to the dependency manager?

          Show
          Knut Anders Hatlen added a comment - With the current architecture, is it possible to make DROP TABLE also drop dependent Lucene indexes? Or would the luceneSupport tool have to be moved into the engine in order to have access to the dependency manager?
          Hide
          Knut Anders Hatlen added a comment -

          Would it be difficult to make the luceneSupport tool capable of using custom analyzers in jar files installed with SQLJ.INSTALL_JAR? I get a ClassNotFoundException if I try that now.

          Show
          Knut Anders Hatlen added a comment - Would it be difficult to make the luceneSupport tool capable of using custom analyzers in jar files installed with SQLJ.INSTALL_JAR? I get a ClassNotFoundException if I try that now.
          Hide
          Knut Anders Hatlen added a comment -

          Is there a way to create multiple Lucene indexes on one column? I didn't find a way to do that currently. It would be useful in the case where you want to use multiple analyzers on the same body of text. For example, if you store Java source files in a CLOB, you might want one index for the entire source file, one for the names of the methods declared in the file, one for searching the comments only, and so on.

          Show
          Knut Anders Hatlen added a comment - Is there a way to create multiple Lucene indexes on one column? I didn't find a way to do that currently. It would be useful in the case where you want to use multiple analyzers on the same body of text. For example, if you store Java source files in a CLOB, you might want one index for the entire source file, one for the names of the methods declared in the file, one for searching the comments only, and so on.
          Hide
          ASF subversion and git services added a comment -

          Commit 1596552 from Knut Anders Hatlen in branch 'code/trunk'
          [ https://svn.apache.org/r1596552 ]

          DERBY-590: Remove handling of impossible exceptions

          Show
          ASF subversion and git services added a comment - Commit 1596552 from Knut Anders Hatlen in branch 'code/trunk' [ https://svn.apache.org/r1596552 ] DERBY-590 : Remove handling of impossible exceptions
          Hide
          Rick Hillegas added a comment -

          Thanks for that cleanup, Knut!

          Show
          Rick Hillegas added a comment - Thanks for that cleanup, Knut!
          Hide
          Knut Anders Hatlen added a comment -

          It looks like most of the methods that call AccessController.doPrivileged() are declared to throw more exceptions than they actually can throw. They are declared to throw all checked exceptions that the privileged action body can throw, in addition to PrivilegedActionException. Since doPrivileged() wraps all checked exceptions thrown by the action body in a PrivilegedActionException, only the PrivilegedActionException can be seen by the callers, unless it is unwrapped first.

          The attached patch exceptions.diff cleans up the exception handling in the following ways:

          1) In the cases where the action body doesn't throw any checked exception, use PrivilegedAction instead of PrivilegedExceptionAction, and remove the impossible exceptions from the throws declaration.

          2) If the action body only throws a single checked exception, unwrap the PrivilegedActionException and re-throw the original exception. Remove PrivilegedActionException from the signature.

          3) If the action body can throw many checked exception types, leave it as it is, but remove all exceptions except PrivilegedActionException from the throws clause in the enclosing method, since that's the only exception that could come out of it.

          4) Remove catch blocks that check for these impossible exceptions higher up in the call hierarchy.

          All regression tests passed.

          Show
          Knut Anders Hatlen added a comment - It looks like most of the methods that call AccessController.doPrivileged() are declared to throw more exceptions than they actually can throw. They are declared to throw all checked exceptions that the privileged action body can throw, in addition to PrivilegedActionException. Since doPrivileged() wraps all checked exceptions thrown by the action body in a PrivilegedActionException, only the PrivilegedActionException can be seen by the callers, unless it is unwrapped first. The attached patch exceptions.diff cleans up the exception handling in the following ways: 1) In the cases where the action body doesn't throw any checked exception, use PrivilegedAction instead of PrivilegedExceptionAction, and remove the impossible exceptions from the throws declaration. 2) If the action body only throws a single checked exception, unwrap the PrivilegedActionException and re-throw the original exception. Remove PrivilegedActionException from the signature. 3) If the action body can throw many checked exception types, leave it as it is, but remove all exceptions except PrivilegedActionException from the throws clause in the enclosing method, since that's the only exception that could come out of it. 4) Remove catch blocks that check for these impossible exceptions higher up in the call hierarchy. All regression tests passed.
          Hide
          ASF subversion and git services added a comment -

          Commit 1596513 from Knut Anders Hatlen in branch 'code/trunk'
          [ https://svn.apache.org/r1596513 ]

          DERBY-590: Update the NetBeans project to use the new names of the Lucene jars

          Show
          ASF subversion and git services added a comment - Commit 1596513 from Knut Anders Hatlen in branch 'code/trunk' [ https://svn.apache.org/r1596513 ] DERBY-590 : Update the NetBeans project to use the new names of the Lucene jars
          Hide
          Knut Anders Hatlen added a comment -

          netbeans2.diff updates the NetBeans project with the new names of the Lucene jar files.

          Show
          Knut Anders Hatlen added a comment - netbeans2.diff updates the NetBeans project with the new names of the Lucene jar files.
          Hide
          Rick Hillegas added a comment -

          Attaching a 3rd rev of the functional spec. This rev clarifies the meaning of the scoreCeiling argument to the Lucene query table function.

          Show
          Rick Hillegas added a comment - Attaching a 3rd rev of the functional spec. This rev clarifies the meaning of the scoreCeiling argument to the Lucene query table function.
          Hide
          ASF subversion and git services added a comment -

          Commit 1593701 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1593701 ]

          DERBY-590: Change the special NO_CEILING value from 0 to NULL for the scoreCeiling argument to the Lucene query table function; commit derby-590-30-aa-nullableScoreCeiling.diff.

          Show
          ASF subversion and git services added a comment - Commit 1593701 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1593701 ] DERBY-590 : Change the special NO_CEILING value from 0 to NULL for the scoreCeiling argument to the Lucene query table function; commit derby-590-30-aa-nullableScoreCeiling.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-30-aa-nullableScoreCeiling.diff. This patch addresses a problem discovered by Kim: the scoreCeiling argument to the Lucene query table function can be an arbitrary number. The special value which means "return all results" should be NULL rather than 0.

          Touches the following files:

          ---------------------

          M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java

          The change.

          ---------------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneBackupTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneInMemoryTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java

          Adjust the tests.

          Show
          Rick Hillegas added a comment - Attaching derby-590-30-aa-nullableScoreCeiling.diff. This patch addresses a problem discovered by Kim: the scoreCeiling argument to the Lucene query table function can be an arbitrary number. The special value which means "return all results" should be NULL rather than 0. Touches the following files: --------------------- M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java The change. --------------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneBackupTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneInMemoryTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java Adjust the tests.
          Hide
          ASF subversion and git services added a comment -

          Commit 1592543 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1592543 ]

          DERBY-590: Change the checked-in version of Lucene from 4.5.0 to 4.7.1; commit derby-590-29-aa-useLucene_4.7.1.diff.

          Show
          ASF subversion and git services added a comment - Commit 1592543 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1592543 ] DERBY-590 : Change the checked-in version of Lucene from 4.5.0 to 4.7.1; commit derby-590-29-aa-useLucene_4.7.1.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-29-aa-useLucene_4.7.1.diff. This patch changes the version of Lucene which is checked into the Derby source tree. This patch changes the version from 4.5.0 to 4.7.1.

          I have run the LuceneSuite using both 4.5.0 and 4.7.1.

          Touches the following files:

          ----------------

          M tools/java/lucene-analyzers-common.jar
          M tools/java/lucene-queryparser.jar
          M tools/java/lucene-core.jar

          The new 4.7.1 jar files.

          ----------------

          M java/optional/org/apache/derby/optional/lucene/DerbyLuceneDir.java

          4.7.1 added some new abstract methods to the Lucene Directory interface. Now DerbyLuceneDir implements them.

          ----------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java

          Changed the expected version string from LUCENE_45 to LUCENE_47.

          Show
          Rick Hillegas added a comment - Attaching derby-590-29-aa-useLucene_4.7.1.diff. This patch changes the version of Lucene which is checked into the Derby source tree. This patch changes the version from 4.5.0 to 4.7.1. I have run the LuceneSuite using both 4.5.0 and 4.7.1. Touches the following files: ---------------- M tools/java/lucene-analyzers-common.jar M tools/java/lucene-queryparser.jar M tools/java/lucene-core.jar The new 4.7.1 jar files. ---------------- M java/optional/org/apache/derby/optional/lucene/DerbyLuceneDir.java 4.7.1 added some new abstract methods to the Lucene Directory interface. Now DerbyLuceneDir implements them. ---------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java Changed the expected version string from LUCENE_45 to LUCENE_47.
          Hide
          ASF subversion and git services added a comment -

          Commit 1592527 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1592527 ]

          DERBY-590: Remove the version ids from the names of the Lucene jar files in order to make it easier to upgrade the version of Lucene checked into the Derby source tree; commit derby-590-28-renameLuceneJars.diff.

          Show
          ASF subversion and git services added a comment - Commit 1592527 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1592527 ] DERBY-590 : Remove the version ids from the names of the Lucene jar files in order to make it easier to upgrade the version of Lucene checked into the Derby source tree; commit derby-590-28-renameLuceneJars.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-28-renameLuceneJars.diff. This patch removes the version ids from the names of the Lucene jar files. This should make it easier to upgrade the version of Lucene which is checked into the Derby codeline.

          After I commit this patch, developers will need to adjust the classpath which they use for the tests.

          Touches the following files:

          A + tools/java/lucene-analyzers-common.jar
          D tools/java/lucene-analyzers-common-4.5.0.jar
          A + tools/java/lucene-queryparser.jar
          D tools/java/lucene-queryparser-4.5.0.jar
          A + tools/java/lucene-core.jar
          D tools/java/lucene-core-4.5.0.jar
          M tools/ant/properties/extrapath.properties

          Show
          Rick Hillegas added a comment - Attaching derby-590-28-renameLuceneJars.diff. This patch removes the version ids from the names of the Lucene jar files. This should make it easier to upgrade the version of Lucene which is checked into the Derby codeline. After I commit this patch, developers will need to adjust the classpath which they use for the tests. Touches the following files: A + tools/java/lucene-analyzers-common.jar D tools/java/lucene-analyzers-common-4.5.0.jar A + tools/java/lucene-queryparser.jar D tools/java/lucene-queryparser-4.5.0.jar A + tools/java/lucene-core.jar D tools/java/lucene-core-4.5.0.jar M tools/ant/properties/extrapath.properties
          Hide
          Rick Hillegas added a comment -

          Attaching a second rev of the functional spec.

          Show
          Rick Hillegas added a comment - Attaching a second rev of the functional spec.
          Hide
          ASF subversion and git services added a comment -

          Commit 1592214 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1592214 ]

          DERBY-590: Add LuceneUtils to the public api; commit derby-590-27-aa-publicAPILuceneUtils.diff.

          Show
          ASF subversion and git services added a comment - Commit 1592214 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1592214 ] DERBY-590 : Add LuceneUtils to the public api; commit derby-590-27-aa-publicAPILuceneUtils.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-27-aa-publicAPILuceneUtils.diff. This patch adds org.apache.derby.optional.api.LuceneUtils to the public api. This class contains utility methods for use in constructing Lucene Analyzers (for indexing data) and Lucene QueryParsers (for parsing queries). The methods are described in the functional spec.

          Support has been added for in-memory databases and for backup/restore. Logic has been put in to prevent customers from using the Lucene plugin on encrypted databases, where the plugin could leak sensitive data. At this point, I believe that we can expose the plugin as an experimental tool in release 10.11. So it is time to expose LuceneUtils in the public api.

          I need to add a section to the functional spec, setting users' expectations about schema evolution, concurrency, and the freshness of Lucene results. After that, we should be ready to document this experimental tool.

          Touches the following files:

          M build.xml
          M tools/javadoc/publishedapi.ant

          Show
          Rick Hillegas added a comment - Attaching derby-590-27-aa-publicAPILuceneUtils.diff. This patch adds org.apache.derby.optional.api.LuceneUtils to the public api. This class contains utility methods for use in constructing Lucene Analyzers (for indexing data) and Lucene QueryParsers (for parsing queries). The methods are described in the functional spec. Support has been added for in-memory databases and for backup/restore. Logic has been put in to prevent customers from using the Lucene plugin on encrypted databases, where the plugin could leak sensitive data. At this point, I believe that we can expose the plugin as an experimental tool in release 10.11. So it is time to expose LuceneUtils in the public api. I need to add a section to the functional spec, setting users' expectations about schema evolution, concurrency, and the freshness of Lucene results. After that, we should be ready to document this experimental tool. Touches the following files: M build.xml M tools/javadoc/publishedapi.ant
          Hide
          ASF subversion and git services added a comment -

          Commit 1591910 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1591910 ]

          DERBY-590: Add backup/restore support for Lucene indexes and prevent the Lucene plugin from being used in an encrypted database; commit derby-590-26-ad-backupRestoreEncryption.diff.

          Show
          ASF subversion and git services added a comment - Commit 1591910 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1591910 ] DERBY-590 : Add backup/restore support for Lucene indexes and prevent the Lucene plugin from being used in an encrypted database; commit derby-590-26-ad-backupRestoreEncryption.diff.
          Hide
          ASF subversion and git services added a comment -

          Commit 1591839 from Knut Anders Hatlen in branch 'code/trunk'
          [ https://svn.apache.org/r1591839 ]

          DERBY-590: Make the NetBeans project recognize the Lucene classes

          Show
          ASF subversion and git services added a comment - Commit 1591839 from Knut Anders Hatlen in branch 'code/trunk' [ https://svn.apache.org/r1591839 ] DERBY-590 : Make the NetBeans project recognize the Lucene classes
          Hide
          Knut Anders Hatlen added a comment -

          The attached patch, netbeans.diff, makes the NetBeans project recognize the new source directory (java/optional), and makes it look into the tools/java/lucene*.jar files to help with code completion for the Lucene API calls.

          The patch modifies tools/ide/netbeans/nbproject/project.xml.

          Show
          Knut Anders Hatlen added a comment - The attached patch, netbeans.diff, makes the NetBeans project recognize the new source directory (java/optional), and makes it look into the tools/java/lucene*.jar files to help with code completion for the Lucene API calls. The patch modifies tools/ide/netbeans/nbproject/project.xml.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-26-ad-backupRestoreEncryption.diff. This patch enforces the incompatiblity of encryption with the Lucene plugin.

          I can't think of a good way to support Derby encryption side-by-side with the performance-driven need for random access to the Lucene indexes. So I have put in logic to enforce the following:

          1) If the plugin has been loaded, then you can't encrypt the database.

          2) If the database is encrypted, then you can't load the plugin.

          The user documentation will need to state that you should use an encrypted file system if you need Lucene indexes plus the protection of an encrypted database.

          Touches the following additional files:

          ----------

          M java/engine/org/apache/derby/loc/messages.xml
          M java/shared/org/apache/derby/shared/common/reference/SQLState.java

          Adds a new error message which is raised when you try to encrypt a database loaded with the plugin or when you try to load the plugin into an encrypted database.

          ----------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java

          New tests.

          Show
          Rick Hillegas added a comment - Attaching derby-590-26-ad-backupRestoreEncryption.diff. This patch enforces the incompatiblity of encryption with the Lucene plugin. I can't think of a good way to support Derby encryption side-by-side with the performance-driven need for random access to the Lucene indexes. So I have put in logic to enforce the following: 1) If the plugin has been loaded, then you can't encrypt the database. 2) If the database is encrypted, then you can't load the plugin. The user documentation will need to state that you should use an encrypted file system if you need Lucene indexes plus the protection of an encrypted database. Touches the following additional files: ---------- M java/engine/org/apache/derby/loc/messages.xml M java/shared/org/apache/derby/shared/common/reference/SQLState.java Adds a new error message which is raised when you try to encrypt a database loaded with the plugin or when you try to load the plugin into an encrypted database. ---------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java New tests.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-26-ac-backupRestore.diff. This patch adds backup/restore support for Lucene indexes. I will run regression tests. I will also hold off committing this patch for a couple days to give people an opportunity to suggest alternative approaches.

          Code is added to BasicDatabase to perform the backup. The backup() methods in BasicDatabase check to see if the database's top level directory contains a LUCENE subdirectory. If so, the LUCENE directory is recursively copied to the backup directory.

          Similarly, code is added to BaseDataFileFactory to perform the restoration. The loop which checks for seg* subdirectories now checks to see if the backup contains a LUCENE subdirectory. If so, it is recursively copied to the restored location.

          I am happy to move this code elsewhere if people think there is an architecturally better spot.

          Touches the following files:

          ---------------

          M java/engine/org/apache/derby/iapi/util/StringUtil.java

          Encapsulated a scrap of code which takes the canonical name of the database and extracts the database directory name from it. This method is now used both by the RawStore backup code to figure out where to copy the seg* files and this method is used by BasicDatabase to figure out where to copy the LUCENE directory.

          I'm open to suggestions for a better place to put this code. Nothing better jumped out at me.

          ---------------

          M java/engine/org/apache/derby/impl/db/BasicDatabase.java

          The new code to backup the LUCENE directory.

          ---------------

          M java/engine/org/apache/derby/impl/io/vfmem/DataStore.java

          This fixes a bug in the in-memory storage factory. When asked to list its child directories, an in-memory directory used to list its grandchildren and all other descendants. This has been fixed so that only the child directories are returned.

          ---------------

          M java/engine/org/apache/derby/impl/store/raw/RawStore.java

          Amended to use the shared method which extracts the database directory name from the canonical name.

          ---------------

          M java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java

          Adds logic to restore the LUCENE directory.

          ---------------

          M java/engine/org/apache/derby/database/Database.java

          The constant identifying the name of the LUCENE directory is moved to this interface so that it can be used by BasicDatabase.

          ---------------

          M java/optional/org/apache/derby/optional/lucene/DerbyLuceneDir.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          M java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java

          I reworked how the Lucene index path is recursively created.

          ---------------

          A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneBackupTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java

          Added a test for backup/restore of Lucene indexes.

          Show
          Rick Hillegas added a comment - Attaching derby-590-26-ac-backupRestore.diff. This patch adds backup/restore support for Lucene indexes. I will run regression tests. I will also hold off committing this patch for a couple days to give people an opportunity to suggest alternative approaches. Code is added to BasicDatabase to perform the backup. The backup() methods in BasicDatabase check to see if the database's top level directory contains a LUCENE subdirectory. If so, the LUCENE directory is recursively copied to the backup directory. Similarly, code is added to BaseDataFileFactory to perform the restoration. The loop which checks for seg* subdirectories now checks to see if the backup contains a LUCENE subdirectory. If so, it is recursively copied to the restored location. I am happy to move this code elsewhere if people think there is an architecturally better spot. Touches the following files: --------------- M java/engine/org/apache/derby/iapi/util/StringUtil.java Encapsulated a scrap of code which takes the canonical name of the database and extracts the database directory name from it. This method is now used both by the RawStore backup code to figure out where to copy the seg* files and this method is used by BasicDatabase to figure out where to copy the LUCENE directory. I'm open to suggestions for a better place to put this code. Nothing better jumped out at me. --------------- M java/engine/org/apache/derby/impl/db/BasicDatabase.java The new code to backup the LUCENE directory. --------------- M java/engine/org/apache/derby/impl/io/vfmem/DataStore.java This fixes a bug in the in-memory storage factory. When asked to list its child directories, an in-memory directory used to list its grandchildren and all other descendants. This has been fixed so that only the child directories are returned. --------------- M java/engine/org/apache/derby/impl/store/raw/RawStore.java Amended to use the shared method which extracts the database directory name from the canonical name. --------------- M java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java Adds logic to restore the LUCENE directory. --------------- M java/engine/org/apache/derby/database/Database.java The constant identifying the name of the LUCENE directory is moved to this interface so that it can be used by BasicDatabase. --------------- M java/optional/org/apache/derby/optional/lucene/DerbyLuceneDir.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java M java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java I reworked how the Lucene index path is recursively created. --------------- A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneBackupTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java Added a test for backup/restore of Lucene indexes.
          Hide
          ASF subversion and git services added a comment -

          Commit 1589124 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1589124 ]

          DERBY-590: Add support for creating Lucene indexes in in-memory databases; commit derby-590-24-ad-luceneDirectory.diff.

          Show
          ASF subversion and git services added a comment - Commit 1589124 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1589124 ] DERBY-590 : Add support for creating Lucene indexes in in-memory databases; commit derby-590-24-ad-luceneDirectory.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-24-ad-luceneDirectory.diff. This patch adds support for creating Lucene indexes in in-memory databases.

          1) This patch supplies an implementation of a Lucene Directory backed by a Derby StorageFactory.

          2) This patch supplies implementations of Lucene IndexInput and IndexOutput which are backed by the random-access file abstractions obtained from Derby StorageFiles.

          Touches the following files:

          -------------------------

          M java/engine/org/apache/derby/impl/io/DirRandomAccessFile.java
          M java/engine/org/apache/derby/impl/io/vfmem/VirtualRandomAccessFile.java
          M java/engine/org/apache/derby/io/StorageRandomAccessFile.java
          M java/testing/org/apache/derbyTesting/functionTests/util/corruptio/CorruptRandomAccessFile.java

          Adds a clone() method to StorageRandomAccessFile and its implementations in order to support the cloning of DerbyIndexInputs.

          -------------------------

          A java/optional/org/apache/derby/optional/lucene/DerbyLuceneDir.java

          Derby implementation of a Lucene Directory.

          -------------------------

          A java/optional/org/apache/derby/optional/lucene/DerbyIndexInput.java

          Derby implementation of a Lucene IndexInput.

          -------------------------

          A java/optional/org/apache/derby/optional/lucene/DerbyIndexOutput.java

          Derby implementation of a Lucene IndexOutput.

          -------------------------

          M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          M java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java

          These classes are updated to use the new StorageFactory-based abstractions.

          -------------------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneInMemoryTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java

          Adds a new test for in-memory Lucene indexes.

          Show
          Rick Hillegas added a comment - Attaching derby-590-24-ad-luceneDirectory.diff. This patch adds support for creating Lucene indexes in in-memory databases. 1) This patch supplies an implementation of a Lucene Directory backed by a Derby StorageFactory. 2) This patch supplies implementations of Lucene IndexInput and IndexOutput which are backed by the random-access file abstractions obtained from Derby StorageFiles. Touches the following files: ------------------------- M java/engine/org/apache/derby/impl/io/DirRandomAccessFile.java M java/engine/org/apache/derby/impl/io/vfmem/VirtualRandomAccessFile.java M java/engine/org/apache/derby/io/StorageRandomAccessFile.java M java/testing/org/apache/derbyTesting/functionTests/util/corruptio/CorruptRandomAccessFile.java Adds a clone() method to StorageRandomAccessFile and its implementations in order to support the cloning of DerbyIndexInputs. ------------------------- A java/optional/org/apache/derby/optional/lucene/DerbyLuceneDir.java Derby implementation of a Lucene Directory. ------------------------- A java/optional/org/apache/derby/optional/lucene/DerbyIndexInput.java Derby implementation of a Lucene IndexInput. ------------------------- A java/optional/org/apache/derby/optional/lucene/DerbyIndexOutput.java Derby implementation of a Lucene IndexOutput. ------------------------- M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java M java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java These classes are updated to use the new StorageFactory-based abstractions. ------------------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneInMemoryTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java Adds a new test for in-memory Lucene indexes.
          Hide
          Rick Hillegas added a comment -

          Attaching a first rev of a functional spec for this feature. I welcome the community's suggestions for how to improve this feature and its api. You will notice that the Lucene plugin is language-sensitive, providing support for several languages understood by active Derby contributors, such as Dutch, French, German, Hindi, Norwegian, Spanish, and Swedish. Your impressions about the quality of that support would be appreciated too.

          Show
          Rick Hillegas added a comment - Attaching a first rev of a functional spec for this feature. I welcome the community's suggestions for how to improve this feature and its api. You will notice that the Lucene plugin is language-sensitive, providing support for several languages understood by active Derby contributors, such as Dutch, French, German, Hindi, Norwegian, Spanish, and Swedish. Your impressions about the quality of that support would be appreciated too.
          Hide
          ASF subversion and git services added a comment -

          Commit 1586755 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1586755 ]

          DERBY-590: Fix locale-related errors which surfaced on some platforms during the Lucene tests; commit derby-590-23-aa-correctTestLocale.diff.

          Show
          ASF subversion and git services added a comment - Commit 1586755 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1586755 ] DERBY-590 : Fix locale-related errors which surfaced on some platforms during the Lucene tests; commit derby-590-23-aa-correctTestLocale.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-23-aa-correctTestLocale.diff. This patch attempts to fix the platform-specific failures in the Lucene tests described on DERBY-6539.

          The problem arose because the tests were using a Swedish rather than an English locale. The patch makes the following changes:

          1) Forces LuceneSupportTest to use an English locale.

          2) Adjusts the order in which the decorators compose for LuceneSupportPermsTest. Even though the test thought that an English locale was being forced, the decorator machinery silently failed to install the English locale and the test reverted to the machine's default locale (Swedish in this case).

          Adjusting the order in which decorators compose may produce other errors in the tests. But this patch runs cleanly both on the problem machine and on my own machine.

          Touches the following files:

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-23-aa-correctTestLocale.diff. This patch attempts to fix the platform-specific failures in the Lucene tests described on DERBY-6539 . The problem arose because the tests were using a Swedish rather than an English locale. The patch makes the following changes: 1) Forces LuceneSupportTest to use an English locale. 2) Adjusts the order in which the decorators compose for LuceneSupportPermsTest. Even though the test thought that an English locale was being forced, the decorator machinery silently failed to install the English locale and the test reverted to the machine's default locale (Swedish in this case). Adjusting the order in which decorators compose may produce other errors in the tests. But this patch runs cleanly both on the problem machine and on my own machine. Touches the following files: M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java
          Hide
          ASF subversion and git services added a comment -

          Commit 1586114 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1586114 ]

          DERBY-590: Hide doPrivileged() blocks inside private methods, seal derbyoptionaltools.jar, and create more relevant SQLExceptions; commit derby-590-22-aa-cleanupPrivacy.diff.

          Show
          ASF subversion and git services added a comment - Commit 1586114 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1586114 ] DERBY-590 : Hide doPrivileged() blocks inside private methods, seal derbyoptionaltools.jar, and create more relevant SQLExceptions; commit derby-590-22-aa-cleanupPrivacy.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-22-aa-cleanupPrivacy.diff. This patch addresses Knut's suggestions:

          1) Makes all of the privileged methods private. Mostly, this didn't cause any disruption. For a couple methods, I created package-visible methods to do the heavy lifting outside a doPrivileged block and then I called those unprotected methods from private doPrivileged blocks.

          2) Seals derbyoptionaltools.jar.

          3) Uses PublicAPI.wrapStandardException() to construct SQLExceptions from StandardExceptions.

          Touches the following files:

          M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          M java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java
          M build.xml

          Show
          Rick Hillegas added a comment - Attaching derby-590-22-aa-cleanupPrivacy.diff. This patch addresses Knut's suggestions: 1) Makes all of the privileged methods private. Mostly, this didn't cause any disruption. For a couple methods, I created package-visible methods to do the heavy lifting outside a doPrivileged block and then I called those unprotected methods from private doPrivileged blocks. 2) Seals derbyoptionaltools.jar. 3) Uses PublicAPI.wrapStandardException() to construct SQLExceptions from StandardExceptions. Touches the following files: M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java M java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java M build.xml
          Hide
          ASF subversion and git services added a comment -

          Commit 1585774 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1585774 ]

          DERBY-590: Prevent Derby from using an earlier version of Lucene to read an index created by a later version of Lucene; commit derby-590-21-aa-noTimeTravel.diff.

          Show
          ASF subversion and git services added a comment - Commit 1585774 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1585774 ] DERBY-590 : Prevent Derby from using an earlier version of Lucene to read an index created by a later version of Lucene; commit derby-590-21-aa-noTimeTravel.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-21-aa-noTimeTravel.diff. This patch adds a check which prevents you from using an earlier version of Lucene to read an index created by a later version of Lucene.

          It's my impression that Lucene makes weaker backward compatibility guarantees than Derby does. And there are few products which support this kind of time travel. I think it is safest to disallow this up front. If we find cases in which this limitation can be relaxed, then we can consider loosening this restriction later on.

          This patch also adds a new testing switch. By default, Lucene plugin test results hardwire the expected version number as 4.5.0. However, by setting the following flag, you can tell the tests that you are using a different Lucene version...

          -Dderby.tests.lucene.version=$luceneVersionName

          ...where $luceneVersionName is a string like "4.5.0" or "4.7.1".

          Touches the following files:

          -------------

          M java/engine/org/apache/derby/loc/messages.xml
          M java/shared/org/apache/derby/shared/common/reference/SQLState.java
          M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java

          New error message and new check to verify that time travel isn't allowed.

          -------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java

          Adds support for new derby.tests.lucene.version knob.

          Show
          Rick Hillegas added a comment - Attaching derby-590-21-aa-noTimeTravel.diff. This patch adds a check which prevents you from using an earlier version of Lucene to read an index created by a later version of Lucene. It's my impression that Lucene makes weaker backward compatibility guarantees than Derby does. And there are few products which support this kind of time travel. I think it is safest to disallow this up front. If we find cases in which this limitation can be relaxed, then we can consider loosening this restriction later on. This patch also adds a new testing switch. By default, Lucene plugin test results hardwire the expected version number as 4.5.0. However, by setting the following flag, you can tell the tests that you are using a different Lucene version... -Dderby.tests.lucene.version=$luceneVersionName ...where $luceneVersionName is a string like "4.5.0" or "4.7.1". Touches the following files: ------------- M java/engine/org/apache/derby/loc/messages.xml M java/shared/org/apache/derby/shared/common/reference/SQLState.java M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java New error message and new check to verify that time travel isn't allowed. ------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java Adds support for new derby.tests.lucene.version knob.
          Hide
          Rick Hillegas added a comment -

          Thanks, Knut. That's a great idea.

          Show
          Rick Hillegas added a comment - Thanks, Knut. That's a great idea.
          Hide
          Knut Anders Hatlen added a comment -

          We might want to change LuceneSupport.sqlException(StandardException) to call PublicAPI.wrapStandardException() instead of invoking the SQLException constructor directly. The advantages are that the PublicAPI wrapper method will use the correct subclass of SQLException, if appropriate, and it will also wrap the exception in a way that's recognized by error handlers higher up, which reduces the amount of additional wrapping.

          For example, an exception currently reported like this

          ij> call LuceneSupport.createIndex('T', 'titles', 'title', null);
          ERROR 38000: The exception 'java.sql.SQLException: The schema, table or column does not exist or the column is not a string type.' was thrown while evaluating an expression.
          ERROR 42XBA: The schema, table or column does not exist or the column is not a string type.
          

          will instead be reported like this

          ij> call LuceneSupport.createIndex('T', 'titles', 'title', null);
          ERROR 42XBA: The schema, table or column does not exist or the column is not a string type.
          
          Show
          Knut Anders Hatlen added a comment - We might want to change LuceneSupport.sqlException(StandardException) to call PublicAPI.wrapStandardException() instead of invoking the SQLException constructor directly. The advantages are that the PublicAPI wrapper method will use the correct subclass of SQLException, if appropriate, and it will also wrap the exception in a way that's recognized by error handlers higher up, which reduces the amount of additional wrapping. For example, an exception currently reported like this ij> call LuceneSupport.createIndex('T', 'titles', 'title', null); ERROR 38000: The exception 'java.sql.SQLException: The schema, table or column does not exist or the column is not a string type.' was thrown while evaluating an expression. ERROR 42XBA: The schema, table or column does not exist or the column is not a string type. will instead be reported like this ij> call LuceneSupport.createIndex('T', 'titles', 'title', null); ERROR 42XBA: The schema, table or column does not exist or the column is not a string type.
          Hide
          Knut Anders Hatlen added a comment -

          The LuceneSupport.getLastModified() method is public, which is unfortunate since it allows unprivileged code to run a privileged operation on arbitrary files using derbyoptionaltools.jar's privileges. See Dan's veto in DERBY-3745 for details.

          I don't see any callers of that method, though, so the easiest solution is probably to remove it.

          There are also similar methods with package visibility (deleteFile(), isDirectory(), listFiles(), fileExists(), getIndexReader(), getAnalyzer()). That might be okay, if derbyoptionaltools.jar's manifest seals the org.apache.derby.optional.lucene package, but I think it would be better if the visibility of those methods were reduced further.

          Show
          Knut Anders Hatlen added a comment - The LuceneSupport.getLastModified() method is public, which is unfortunate since it allows unprivileged code to run a privileged operation on arbitrary files using derbyoptionaltools.jar's privileges. See Dan's veto in DERBY-3745 for details. I don't see any callers of that method, though, so the easiest solution is probably to remove it. There are also similar methods with package visibility (deleteFile(), isDirectory(), listFiles(), fileExists(), getIndexReader(), getAnalyzer()). That might be okay, if derbyoptionaltools.jar's manifest seals the org.apache.derby.optional.lucene package, but I think it would be better if the visibility of those methods were reduced further.
          Hide
          ASF subversion and git services added a comment -

          Commit 1585488 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1585488 ]

          DERBY-590: Add the ability to configure the query parser used for lucene queries; commit derby-590-20-aa-customQueryParser.diff.

          Show
          ASF subversion and git services added a comment - Commit 1585488 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1585488 ] DERBY-590 : Add the ability to configure the query parser used for lucene queries; commit derby-590-20-aa-customQueryParser.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-20-aa-customQueryParser.diff. This patch adds another parameter to the table function which performs the Lucene search. The new argument is the name of a static, public method which creates a query parser.

          Lucene lets you extend its query language or even replace its query language with your own, custom language.

          The table function relies on two classes for which Lucene supplies multiple implementations:

          1) Analyzer - This is the locale-specific logic which turns a block of text into a series of indexable terms.

          2) QueryParser - This is the application-specific logic for interpreting the query string.

          As of derby-590-09-aa-localeSensitiveAnalysis.diff, the user can supply their own locale-specific Analyzer. As of this patch, the user can also supply their own custom QueryParser for interpreting query strings.

          Touches the following files:

          -------------

          M java/optional/org/apache/derby/optional/api/LuceneUtils.java
          M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/build.xml
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-20-aa-customQueryParser.diff. This patch adds another parameter to the table function which performs the Lucene search. The new argument is the name of a static, public method which creates a query parser. Lucene lets you extend its query language or even replace its query language with your own, custom language. The table function relies on two classes for which Lucene supplies multiple implementations: 1) Analyzer - This is the locale-specific logic which turns a block of text into a series of indexable terms. 2) QueryParser - This is the application-specific logic for interpreting the query string. As of derby-590-09-aa-localeSensitiveAnalysis.diff, the user can supply their own locale-specific Analyzer. As of this patch, the user can also supply their own custom QueryParser for interpreting query strings. Touches the following files: ------------- M java/optional/org/apache/derby/optional/api/LuceneUtils.java M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/build.xml M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java
          Hide
          ASF subversion and git services added a comment -

          Commit 1585315 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1585315 ]

          DERBY-590: Add a windowSize parameter to the lucene query function; commit derby-590-19-aa-cleanupAPI2.diff.

          Show
          ASF subversion and git services added a comment - Commit 1585315 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1585315 ] DERBY-590 : Add a windowSize parameter to the lucene query function; commit derby-590-19-aa-cleanupAPI2.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-19-aa-cleanupAPI2.diff. This patch adds another parameter to the query table function used by the plugin.

          Lucene has a concept of a window's worth of results. The Lucene classes allow you to process a series of windows of results by configuring two parameters:

          1) The window size.

          2) The score of the last document in the previous window.

          Right now the plugin allows you to set 2). I think the plugin would be more useful if you could set 1) as well. This would also make it easier to explain how the plugin API maps onto Lucene concepts.

          Technically, neither of these parameters are needed. That is because the user can specify OFFSET and FETCH NEXT clauses to produce the same effect. However, I think that supporting both of these parameters will make some families of queries perform better.

          This patch adds a windowSize parameter to the query table function.

          Touches the following files:

          -------------

          M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-19-aa-cleanupAPI2.diff. This patch adds another parameter to the query table function used by the plugin. Lucene has a concept of a window's worth of results. The Lucene classes allow you to process a series of windows of results by configuring two parameters: 1) The window size. 2) The score of the last document in the previous window. Right now the plugin allows you to set 2). I think the plugin would be more useful if you could set 1) as well. This would also make it easier to explain how the plugin API maps onto Lucene concepts. Technically, neither of these parameters are needed. That is because the user can specify OFFSET and FETCH NEXT clauses to produce the same effect. However, I think that supporting both of these parameters will make some families of queries perform better. This patch adds a windowSize parameter to the query table function. Touches the following files: ------------- M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java
          Hide
          ASF subversion and git services added a comment -

          Commit 1585313 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1585313 ]

          DERBY-590: Rename rank to score and forbid duplicate column names for the Lucene plugin; commit derby-590-18-aa-cleanupAPI.diff.

          Show
          ASF subversion and git services added a comment - Commit 1585313 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1585313 ] DERBY-590 : Rename rank to score and forbid duplicate column names for the Lucene plugin; commit derby-590-18-aa-cleanupAPI.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-18-aa-cleanupAPI.diff. This patch cleans up the plugin api by renaming a column and a parameter. The patch also forbids name conflicts between user-supplied and system-supplied column names.

          While preparing to write a functional spec for the plugin, I tripped across some parts of the api which need some tweaking so that users can understand the tool. This patch is the first set of tweaks.

          The existing plugin uses the term "rank" as a synonym for Lucene's concept of a "score". I think it will be easier to explain the plugin if we stick to Lucene's terms. So this patch renames the following:

          o The "rank" column returned by LuceneQueryVTI has been renamed to "score".

          o The "rankCutoff" argument to LuceneQueryVTI has been renamed to "scoreCeiling".

          While I was in there, I changed the type of scoreCeiling from double to real so that it corresponds with the type of score. I also added some logic to prevent users from creating a LuceneQueryVTI which returns a data set with two columns named "documentID" or two columns named "score".

          Touches the following files:

          -------------

          M java/engine/org/apache/derby/loc/messages.xml
          M java/shared/org/apache/derby/shared/common/reference/SQLState.java

          New error message.

          -------------

          M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java

          Rename columns and forbid name conflicts.

          -------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java

          Adjust existing tests and add new test case for name conflicts.

          Show
          Rick Hillegas added a comment - Attaching derby-590-18-aa-cleanupAPI.diff. This patch cleans up the plugin api by renaming a column and a parameter. The patch also forbids name conflicts between user-supplied and system-supplied column names. While preparing to write a functional spec for the plugin, I tripped across some parts of the api which need some tweaking so that users can understand the tool. This patch is the first set of tweaks. The existing plugin uses the term "rank" as a synonym for Lucene's concept of a "score". I think it will be easier to explain the plugin if we stick to Lucene's terms. So this patch renames the following: o The "rank" column returned by LuceneQueryVTI has been renamed to "score". o The "rankCutoff" argument to LuceneQueryVTI has been renamed to "scoreCeiling". While I was in there, I changed the type of scoreCeiling from double to real so that it corresponds with the type of score. I also added some logic to prevent users from creating a LuceneQueryVTI which returns a data set with two columns named "documentID" or two columns named "score". Touches the following files: ------------- M java/engine/org/apache/derby/loc/messages.xml M java/shared/org/apache/derby/shared/common/reference/SQLState.java New error message. ------------- M java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java Rename columns and forbid name conflicts. ------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java Adjust existing tests and add new test case for name conflicts.
          Hide
          ASF subversion and git services added a comment -

          Commit 1584859 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1584859 ]

          DERBY-590: Eliminate some file-closure race conditions by explicitly closing the stream from which index properties are read; commit derby-590-17-aa-closeInputStreamOnPropertiesFile.diff.

          Show
          ASF subversion and git services added a comment - Commit 1584859 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1584859 ] DERBY-590 : Eliminate some file-closure race conditions by explicitly closing the stream from which index properties are read; commit derby-590-17-aa-closeInputStreamOnPropertiesFile.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-17-aa-closeInputStreamOnPropertiesFile.diff. This patch addresses a file closure bug (DERBY-6536) which was causing errors on Windows platforms.

          As part of dropping an index (or unloading the plugin), we need to delete files and directories. The original code for this deletion never checked the boolean return status of File.delete(). I have added code so that we check that status now and raise an error if the deletion fails.

          Once I made that change, the offending file popped into focus when I ran the test case on DERBY-6536. The file is the derby-lucene.properties file which holds the persistent metadata for the Lucene index. The file was being read through a FileInputStream which was created transiently as an argument to Properties.load. The transient stream was never explicitly closed. Maybe it was being garbage collected (and closed) sooner on non-Windows systems and that's why we only saw this problem on Windows. The code now stores the stream in a variable so that it can be closed explicitly after the properties are loaded. This causes the test case on DERBY-6536 to run cleanly. The whole LuceneSuite now runs cleanly on that Windows platform.

          Touches the following file:

          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-17-aa-closeInputStreamOnPropertiesFile.diff. This patch addresses a file closure bug ( DERBY-6536 ) which was causing errors on Windows platforms. As part of dropping an index (or unloading the plugin), we need to delete files and directories. The original code for this deletion never checked the boolean return status of File.delete(). I have added code so that we check that status now and raise an error if the deletion fails. Once I made that change, the offending file popped into focus when I ran the test case on DERBY-6536 . The file is the derby-lucene.properties file which holds the persistent metadata for the Lucene index. The file was being read through a FileInputStream which was created transiently as an argument to Properties.load. The transient stream was never explicitly closed. Maybe it was being garbage collected (and closed) sooner on non-Windows systems and that's why we only saw this problem on Windows. The code now stores the stream in a variable so that it can be closed explicitly after the properties are loaded. This causes the test case on DERBY-6536 to run cleanly. The whole LuceneSuite now runs cleanly on that Windows platform. Touches the following file: M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          Hide
          Knut Anders Hatlen added a comment -

          The new failure seems to be caused by a data corruption that happens in an old and buggy version and is detected by an assert in the new version. We have some workarounds for this problem in the upgrade tests already. I don't have the code handy right now, but you could search for code comments referencing DERBY-4577 for ideas.

          Show
          Knut Anders Hatlen added a comment - The new failure seems to be caused by a data corruption that happens in an old and buggy version and is detected by an assert in the new version. We have some workarounds for this problem in the upgrade tests already. I don't have the code handy right now, but you could search for code comments referencing DERBY-4577 for ideas.
          Hide
          ASF subversion and git services added a comment -

          Commit 1584721 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1584721 ]

          DERBY-590: Attempt to fix upgrade tests which were broken by derby-590-15-aa-requireHardUpgrade.diff; commit derby-590-16-aa-adjustUpgradeTest.diff.

          Show
          ASF subversion and git services added a comment - Commit 1584721 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1584721 ] DERBY-590 : Attempt to fix upgrade tests which were broken by derby-590-15-aa-requireHardUpgrade.diff; commit derby-590-16-aa-adjustUpgradeTest.diff.
          Hide
          Rick Hillegas added a comment -

          Thanks for that analysis, Knut. Attaching derby-590-16-aa-adjustUpgradeTest.diff. This patch attempts to make the new upgrade test case sensitive to the starting point of the upgrade trajectory.

          I ran the upgrade test with the following starting points:

          10.0.2.1
          10.1.1.0
          10.1.2.1
          10.1.3.1
          10.2.1.6
          10.2.2.0
          10.2.2.1
          10.3.1.4
          10.3.2.1
          10.3.3.0
          10.4.1.3
          10.4.2.0
          10.4.2.1
          10.5.1.1
          10.5.2.0
          10.5.3.0
          10.6.1.0
          10.6.2.1
          10.7.1.1
          10.8.1.2
          10.8.2.2
          10.9.1.0
          10.10.1.1
          10.10.1.2
          

          I no longer see errors in the test case for the Lucene plugin. However, I see the following errors with this set of starting points:

          There were 2 errors:
          1) 10.5.2.0 Upgrade Phase: SOFT UPGRADE java.sql.SQLException: Failed to start database 'singleUse/oneuse62' with class loader sun.misc.Launcher$AppClassLoader@6e0be858, see the next exception for details.
          	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:107)
          	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:133)
          	at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:255)
          	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2841)
          	at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:405)
          	at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:647)
          	at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:301)
          	at org.apache.derby.jdbc.EmbeddedBaseDataSource.getConnection(EmbeddedBaseDataSource.java:618)
          	at org.apache.derby.jdbc.EmbeddedBaseDataSource.getConnection(EmbeddedBaseDataSource.java:555)
          	at org.apache.derbyTesting.functionTests.tests.upgradeTests.PhaseChanger.setUp(PhaseChanger.java:117)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:20)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          Caused by: ERROR XJ040: Failed to start database 'singleUse/oneuse62' with class loader sun.misc.Launcher$AppClassLoader@6e0be858, see the next exception for details.
          	at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:290)
          	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:162)
          	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:74)
          	... 91 more
          Caused by: org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED initSlotTable consistency check failed:  slot 0 minimumRecordSize = 12 totalSpace = 12 recordPortionLength = 12 reservedCount = 0
          	at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162)
          	at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147)
          	at org.apache.derby.impl.store.raw.data.StoredPage.initSlotTable(StoredPage.java:2253)
          	at org.apache.derby.impl.store.raw.data.StoredPage.initFromData(StoredPage.java:851)
          	at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:213)
          	at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
          	at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2540)
          	at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2590)
          	at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319)
          	at org.apache.derby.impl.store.raw.data.StoredPage.getOverflowPage(StoredPage.java:8328)
          	at org.apache.derby.impl.store.raw.data.StoredPage.restoreRecordFromSlot(StoredPage.java:1577)
          	at org.apache.derby.impl.store.raw.data.BasePage.fetchFromSlot(BasePage.java:441)
          	at org.apache.derby.impl.store.raw.data.CachedPage.fetchFromSlot(CachedPage.java:53)
          	at org.apache.derby.impl.store.access.conglomerate.GenericScanController.fetchRows(GenericScanController.java:760)
          	at org.apache.derby.impl.store.access.heap.HeapScan.fetchNext(HeapScan.java:245)
          	at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.clearSPSPlans(DataDictionaryImpl.java:4561)
          	at org.apache.derby.impl.sql.catalog.DD_Version.handleMinorRevisionChange(DD_Version.java:548)
          	at org.apache.derby.impl.sql.catalog.DD_Version.upgradeIfNeeded(DD_Version.java:238)
          	at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.loadDictionaryTables(DataDictionaryImpl.java:7825)
          	at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.boot(DataDictionaryImpl.java:817)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1991)
          	at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:541)
          	at org.apache.derby.impl.services.monitor.FileMonitor.startModule(FileMonitor.java:44)
          	at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:423)
          	at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:196)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1991)
          	at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1819)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1685)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1569)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:988)
          	at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:546)
          	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2802)
          	... 88 more
          2) 10.5.2.0 Upgrade Phase: UPGRADE java.sql.SQLException: Failed to start database 'singleUse/oneuse62' with class loader sun.misc.Launcher$AppClassLoader@6e0be858, see the next exception for details.
          	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:107)
          	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:133)
          	at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:255)
          	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2841)
          	at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:405)
          	at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:647)
          	at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:301)
          	at org.apache.derby.jdbc.EmbeddedBaseDataSource.getConnection(EmbeddedBaseDataSource.java:618)
          	at org.apache.derby.jdbc.EmbeddedBaseDataSource.getConnection(EmbeddedBaseDataSource.java:555)
          	at org.apache.derbyTesting.functionTests.tests.upgradeTests.PhaseChanger.setUp(PhaseChanger.java:117)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:20)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          Caused by: ERROR XJ040: Failed to start database 'singleUse/oneuse62' with class loader sun.misc.Launcher$AppClassLoader@6e0be858, see the next exception for details.
          	at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:290)
          	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:162)
          	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:74)
          	... 91 more
          Caused by: org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED initSlotTable consistency check failed:  slot 0 minimumRecordSize = 12 totalSpace = 12 recordPortionLength = 12 reservedCount = 0
          	at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162)
          	at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147)
          	at org.apache.derby.impl.store.raw.data.StoredPage.initSlotTable(StoredPage.java:2253)
          	at org.apache.derby.impl.store.raw.data.StoredPage.initFromData(StoredPage.java:851)
          	at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:213)
          	at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
          	at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2540)
          	at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2590)
          	at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319)
          	at org.apache.derby.impl.store.raw.data.StoredPage.getOverflowPage(StoredPage.java:8328)
          	at org.apache.derby.impl.store.raw.data.StoredPage.restoreRecordFromSlot(StoredPage.java:1577)
          	at org.apache.derby.impl.store.raw.data.BasePage.fetchFromSlot(BasePage.java:441)
          	at org.apache.derby.impl.store.raw.data.CachedPage.fetchFromSlot(CachedPage.java:53)
          	at org.apache.derby.impl.store.access.conglomerate.GenericScanController.fetchRows(GenericScanController.java:760)
          	at org.apache.derby.impl.store.access.heap.HeapScan.fetchNext(HeapScan.java:245)
          	at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.clearSPSPlans(DataDictionaryImpl.java:4561)
          	at org.apache.derby.impl.sql.catalog.DD_Version.handleMinorRevisionChange(DD_Version.java:548)
          	at org.apache.derby.impl.sql.catalog.DD_Version.upgradeIfNeeded(DD_Version.java:238)
          	at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.loadDictionaryTables(DataDictionaryImpl.java:7825)
          	at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.boot(DataDictionaryImpl.java:817)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1991)
          	at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:541)
          	at org.apache.derby.impl.services.monitor.FileMonitor.startModule(FileMonitor.java:44)
          	at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:423)
          	at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:196)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1991)
          	at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1819)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1685)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1569)
          	at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:988)
          	at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:546)
          	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2802)
          	... 88 more
          There were 3 failures:
          1) testTriggers(org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_7)junit.framework.AssertionFailedError: Column value mismatch @ column '1', row 1:
              Expected: >4<
              Found:    >2<
          	at org.apache.derbyTesting.junit.JDBC.assertRowInResultSet(JDBC.java:1303)
          	at org.apache.derbyTesting.junit.JDBC.assertRowInResultSet(JDBC.java:1215)
          	at org.apache.derbyTesting.junit.JDBC.assertFullResultSetMinion(JDBC.java:1102)
          	at org.apache.derbyTesting.junit.JDBC.assertFullResultSet(JDBC.java:1025)
          	at org.apache.derbyTesting.junit.JDBC.assertFullResultSet(JDBC.java:982)
          	at org.apache.derbyTesting.junit.JDBC.assertFullResultSet(JDBC.java:940)
          	at org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_7.testTriggers(Changes10_7.java:633)
          	at sun.reflect.GeneratedMethodAccessor1023.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:118)
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:440)
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:457)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          2) testDisposableStatisticsExplicit(org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_9)junit.framework.AssertionFailedError: Index statistics for DisposableIndexStatistics tables
          1: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=SQL140404064747790, lcols=1, rows=2000, unique/card=19, created=2014-04-04 06:47:47.874}
          2: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=DUPS_MAIN, lcols=1, rows=2000, unique/card=10, created=2014-04-04 06:47:47.871}
          3: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=SQL140404064747730, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.87}
          4: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=<n/a>, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.875}
          5: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=SQL140404064747760, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.873}
          6: {tableId=d2470e08-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS_FK, indexName=SQL140404064747710, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.728}
          7: {tableId=64418e04-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS_PK_2COL, indexName=SQL140404064747680, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.712}
          8: {tableId=64418e04-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS_PK_2COL, indexName=SQL140404064747680, lcols=2, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.712}
           expected:<7> but was:<8>
          	at org.apache.derbyTesting.functionTests.tests.upgradeTests.helpers.DisposableIndexStatistics.assertStatsCount(DisposableIndexStatistics.java:245)
          	at org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_9.testDisposableStatisticsExplicit(Changes10_9.java:1046)
          	at sun.reflect.GeneratedMethodAccessor1032.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:118)
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:440)
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:457)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          3) testDropTriggerDependencies(org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_11)junit.framework.AssertionFailedError: Expected error(s) ' X0Y25' but no error was thrown.
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementErrorMinion(BaseJDBCTestCase.java:1194)
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:1145)
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:1228)
          	at org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_11.testDropTriggerDependencies(Changes10_11.java:182)
          	at sun.reflect.GeneratedMethodAccessor1039.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:118)
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:440)
          	at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:457)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at org.apache.derbyTesting.junit.BaseTestSetup.run(BasSeTestSetup.java:57)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          	at junit.extensions.TestSetup.run(TestSetup.java:25)
          
          FAILURES!!!
          Tests run: 6504,  Failures: 3,  Errors: 2
          

          These problems don't seem to be related to the fix I'm making, but I could be wrong. The problems seem to be related to triggers and istat statistics. Are these known problems?

          Touches the following file:

          M java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/Changes10_11.java

          Show
          Rick Hillegas added a comment - Thanks for that analysis, Knut. Attaching derby-590-16-aa-adjustUpgradeTest.diff. This patch attempts to make the new upgrade test case sensitive to the starting point of the upgrade trajectory. I ran the upgrade test with the following starting points: 10.0.2.1 10.1.1.0 10.1.2.1 10.1.3.1 10.2.1.6 10.2.2.0 10.2.2.1 10.3.1.4 10.3.2.1 10.3.3.0 10.4.1.3 10.4.2.0 10.4.2.1 10.5.1.1 10.5.2.0 10.5.3.0 10.6.1.0 10.6.2.1 10.7.1.1 10.8.1.2 10.8.2.2 10.9.1.0 10.10.1.1 10.10.1.2 I no longer see errors in the test case for the Lucene plugin. However, I see the following errors with this set of starting points: There were 2 errors: 1) 10.5.2.0 Upgrade Phase: SOFT UPGRADE java.sql.SQLException: Failed to start database 'singleUse/oneuse62' with class loader sun.misc.Launcher$AppClassLoader@6e0be858, see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:107) at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:133) at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:255) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2841) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:405) at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:647) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:301) at org.apache.derby.jdbc.EmbeddedBaseDataSource.getConnection(EmbeddedBaseDataSource.java:618) at org.apache.derby.jdbc.EmbeddedBaseDataSource.getConnection(EmbeddedBaseDataSource.java:555) at org.apache.derbyTesting.functionTests.tests.upgradeTests.PhaseChanger.setUp(PhaseChanger.java:117) at junit.extensions.TestSetup$1.protect(TestSetup.java:20) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) Caused by: ERROR XJ040: Failed to start database 'singleUse/oneuse62' with class loader sun.misc.Launcher$AppClassLoader@6e0be858, see the next exception for details. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:290) at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:162) at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:74) ... 91 more Caused by: org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED initSlotTable consistency check failed: slot 0 minimumRecordSize = 12 totalSpace = 12 recordPortionLength = 12 reservedCount = 0 at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162) at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147) at org.apache.derby.impl.store.raw.data.StoredPage.initSlotTable(StoredPage.java:2253) at org.apache.derby.impl.store.raw.data.StoredPage.initFromData(StoredPage.java:851) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:213) at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295) at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2540) at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2590) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319) at org.apache.derby.impl.store.raw.data.StoredPage.getOverflowPage(StoredPage.java:8328) at org.apache.derby.impl.store.raw.data.StoredPage.restoreRecordFromSlot(StoredPage.java:1577) at org.apache.derby.impl.store.raw.data.BasePage.fetchFromSlot(BasePage.java:441) at org.apache.derby.impl.store.raw.data.CachedPage.fetchFromSlot(CachedPage.java:53) at org.apache.derby.impl.store.access.conglomerate.GenericScanController.fetchRows(GenericScanController.java:760) at org.apache.derby.impl.store.access.heap.HeapScan.fetchNext(HeapScan.java:245) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.clearSPSPlans(DataDictionaryImpl.java:4561) at org.apache.derby.impl.sql.catalog.DD_Version.handleMinorRevisionChange(DD_Version.java:548) at org.apache.derby.impl.sql.catalog.DD_Version.upgradeIfNeeded(DD_Version.java:238) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.loadDictionaryTables(DataDictionaryImpl.java:7825) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.boot(DataDictionaryImpl.java:817) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1991) at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:541) at org.apache.derby.impl.services.monitor.FileMonitor.startModule(FileMonitor.java:44) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:423) at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:196) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1991) at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1819) at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1685) at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1569) at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:988) at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:546) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2802) ... 88 more 2) 10.5.2.0 Upgrade Phase: UPGRADE java.sql.SQLException: Failed to start database 'singleUse/oneuse62' with class loader sun.misc.Launcher$AppClassLoader@6e0be858, see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:107) at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:133) at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:255) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2841) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:405) at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:647) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:301) at org.apache.derby.jdbc.EmbeddedBaseDataSource.getConnection(EmbeddedBaseDataSource.java:618) at org.apache.derby.jdbc.EmbeddedBaseDataSource.getConnection(EmbeddedBaseDataSource.java:555) at org.apache.derbyTesting.functionTests.tests.upgradeTests.PhaseChanger.setUp(PhaseChanger.java:117) at junit.extensions.TestSetup$1.protect(TestSetup.java:20) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) Caused by: ERROR XJ040: Failed to start database 'singleUse/oneuse62' with class loader sun.misc.Launcher$AppClassLoader@6e0be858, see the next exception for details. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:290) at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:162) at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:74) ... 91 more Caused by: org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED initSlotTable consistency check failed: slot 0 minimumRecordSize = 12 totalSpace = 12 recordPortionLength = 12 reservedCount = 0 at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162) at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147) at org.apache.derby.impl.store.raw.data.StoredPage.initSlotTable(StoredPage.java:2253) at org.apache.derby.impl.store.raw.data.StoredPage.initFromData(StoredPage.java:851) at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:213) at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295) at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2540) at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2590) at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319) at org.apache.derby.impl.store.raw.data.StoredPage.getOverflowPage(StoredPage.java:8328) at org.apache.derby.impl.store.raw.data.StoredPage.restoreRecordFromSlot(StoredPage.java:1577) at org.apache.derby.impl.store.raw.data.BasePage.fetchFromSlot(BasePage.java:441) at org.apache.derby.impl.store.raw.data.CachedPage.fetchFromSlot(CachedPage.java:53) at org.apache.derby.impl.store.access.conglomerate.GenericScanController.fetchRows(GenericScanController.java:760) at org.apache.derby.impl.store.access.heap.HeapScan.fetchNext(HeapScan.java:245) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.clearSPSPlans(DataDictionaryImpl.java:4561) at org.apache.derby.impl.sql.catalog.DD_Version.handleMinorRevisionChange(DD_Version.java:548) at org.apache.derby.impl.sql.catalog.DD_Version.upgradeIfNeeded(DD_Version.java:238) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.loadDictionaryTables(DataDictionaryImpl.java:7825) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.boot(DataDictionaryImpl.java:817) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1991) at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:541) at org.apache.derby.impl.services.monitor.FileMonitor.startModule(FileMonitor.java:44) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:423) at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:196) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1991) at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1819) at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1685) at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1569) at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:988) at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:546) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2802) ... 88 more There were 3 failures: 1) testTriggers(org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_7)junit.framework.AssertionFailedError: Column value mismatch @ column '1', row 1: Expected: >4< Found: >2< at org.apache.derbyTesting.junit.JDBC.assertRowInResultSet(JDBC.java:1303) at org.apache.derbyTesting.junit.JDBC.assertRowInResultSet(JDBC.java:1215) at org.apache.derbyTesting.junit.JDBC.assertFullResultSetMinion(JDBC.java:1102) at org.apache.derbyTesting.junit.JDBC.assertFullResultSet(JDBC.java:1025) at org.apache.derbyTesting.junit.JDBC.assertFullResultSet(JDBC.java:982) at org.apache.derbyTesting.junit.JDBC.assertFullResultSet(JDBC.java:940) at org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_7.testTriggers(Changes10_7.java:633) at sun.reflect.GeneratedMethodAccessor1023.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:118) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:440) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:457) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) 2) testDisposableStatisticsExplicit(org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_9)junit.framework.AssertionFailedError: Index statistics for DisposableIndexStatistics tables 1: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=SQL140404064747790, lcols=1, rows=2000, unique/card=19, created=2014-04-04 06:47:47.874} 2: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=DUPS_MAIN, lcols=1, rows=2000, unique/card=10, created=2014-04-04 06:47:47.871} 3: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=SQL140404064747730, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.87} 4: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=<n/a>, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.875} 5: {tableId=40508e0c-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS, indexName=SQL140404064747760, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.873} 6: {tableId=d2470e08-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS_FK, indexName=SQL140404064747710, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.728} 7: {tableId=64418e04-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS_PK_2COL, indexName=SQL140404064747680, lcols=1, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.712} 8: {tableId=64418e04-0145-2cfe-9539-ffffe1d7aa3e, tableName=ISTAT_DISPOSABLE_STATS_PK_2COL, indexName=SQL140404064747680, lcols=2, rows=2000, unique/card=2000, created=2014-04-04 06:47:47.712} expected:<7> but was:<8> at org.apache.derbyTesting.functionTests.tests.upgradeTests.helpers.DisposableIndexStatistics.assertStatsCount(DisposableIndexStatistics.java:245) at org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_9.testDisposableStatisticsExplicit(Changes10_9.java:1046) at sun.reflect.GeneratedMethodAccessor1032.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:118) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:440) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:457) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) 3) testDropTriggerDependencies(org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_11)junit.framework.AssertionFailedError: Expected error(s) ' X0Y25' but no error was thrown. at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementErrorMinion(BaseJDBCTestCase.java:1194) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:1145) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:1228) at org.apache.derbyTesting.functionTests.tests.upgradeTests.Changes10_11.testDropTriggerDependencies(Changes10_11.java:182) at sun.reflect.GeneratedMethodAccessor1039.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:118) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:440) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:457) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BasSeTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) FAILURES!!! Tests run: 6504, Failures: 3, Errors: 2 These problems don't seem to be related to the fix I'm making, but I could be wrong. The problems seem to be related to triggers and istat statistics. Are these known problems? Touches the following file: M java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/Changes10_11.java
          Hide
          Knut Anders Hatlen added a comment -

          I'm seeing failures in the upgrade tests after the latest patch. Also seen in the continuous integration tests: http://download.java.net/javadesktop/derby/request_5589419/

          It looks like it happens when the old version does not have the SYSCS_REGISTER_TOOL procedure. Then it doesn't fail with the expected "Unknown optional tool" error. Instead, it fails with:

          ERROR 42Y03: 'SYSCS_UTIL.SYSCS_REGISTER_TOOL' is not recognized as a function or procedure.

          Or if the old version is so old that it doesn't support the BOOLEAN data type:

          ERROR 42X01: Syntax error: true.

          Show
          Knut Anders Hatlen added a comment - I'm seeing failures in the upgrade tests after the latest patch. Also seen in the continuous integration tests: http://download.java.net/javadesktop/derby/request_5589419/ It looks like it happens when the old version does not have the SYSCS_REGISTER_TOOL procedure. Then it doesn't fail with the expected "Unknown optional tool" error. Instead, it fails with: ERROR 42Y03: 'SYSCS_UTIL.SYSCS_REGISTER_TOOL' is not recognized as a function or procedure. Or if the old version is so old that it doesn't support the BOOLEAN data type: ERROR 42X01: Syntax error: true.
          Hide
          ASF subversion and git services added a comment -

          Commit 1584493 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1584493 ]

          DERBY-590: Require hard-upgrade to 10.11 in order to use the Lucene plugin; commit derby-590-15-aa-requireHardUpgrade.diff.

          Show
          ASF subversion and git services added a comment - Commit 1584493 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1584493 ] DERBY-590 : Require hard-upgrade to 10.11 in order to use the Lucene plugin; commit derby-590-15-aa-requireHardUpgrade.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-15-aa-requireHardUpgrade.diff. This patch adds the requirement that the database be at least at level 10.11 in order to load the Lucene plugin. This prevents users from ending up with stranded Lucene machinery which they can't unload conveniently after soft-downgrade.

          Touches the following files:

          -----------------

          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/Changes10_11.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-15-aa-requireHardUpgrade.diff. This patch adds the requirement that the database be at least at level 10.11 in order to load the Lucene plugin. This prevents users from ending up with stranded Lucene machinery which they can't unload conveniently after soft-downgrade. Touches the following files: ----------------- M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java M java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/Changes10_11.java
          Hide
          ASF subversion and git services added a comment -

          Commit 1584242 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1584242 ]

          DERBY-590: Enforce coarse-grained authorization checks in the Lucene plugin; commit derby-590-14-aa-coarseGrainedAuthorization.diff.

          Show
          ASF subversion and git services added a comment - Commit 1584242 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1584242 ] DERBY-590 : Enforce coarse-grained authorization checks in the Lucene plugin; commit derby-590-14-aa-coarseGrainedAuthorization.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-14-aa-coarseGrainedAuthorization.diff. This patch fixes a bug which allowed read-only users to execute the updateIndex() procedure.

          Touches the following files:

          -----------

          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java

          Add checks for read-only connections.

          -----------

          A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java

          Add tests for using the Lucene pluging with coarse-grained authorization.

          Show
          Rick Hillegas added a comment - Attaching derby-590-14-aa-coarseGrainedAuthorization.diff. This patch fixes a bug which allowed read-only users to execute the updateIndex() procedure. Touches the following files: ----------- M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java Add checks for read-only connections. ----------- A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCoarseAuthorizationTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java Add tests for using the Lucene pluging with coarse-grained authorization.
          Hide
          ASF subversion and git services added a comment -

          Commit 1583855 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1583855 ]

          DERBY-590: Add ability to create a Lucene index on an arbitrary view; commit derby-590-13-aa-indexViews.diff.

          Show
          ASF subversion and git services added a comment - Commit 1583855 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1583855 ] DERBY-590 : Add ability to create a Lucene index on an arbitrary view; commit derby-590-13-aa-indexViews.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-13-aa-indexViews.diff. This patch adds support for creating Lucene indexes on arbitrary views.

          In order to index a view, you must supply a list of columns which will be treated as a key. I could not think of any reason to avoid sharing this capability with tables too. So now you can index a table which doesn't have a primary key. If you omit the key columns, then we try to find a primary key; and we raise an error if we can't find one.

          You do not supply key columns when you update a Lucene index, however. Instead, the plugin looks up the existing key information for the table function.

          Errors can, of course, occur if you drop a key column from the table. If you do this, you will get an error when you try to select from the Lucene index; that is because we still check to see if you have SELECT privilege on the missing column.

          This raises an interesting defect of the plugin: Creating a Lucene index does not prevent you from performing DDL on the table/view which will make it impossible to join the Lucene index back to Derby data. You are on your own there.

          At this point, I think that the api for the plugin is stable enough that I can write a first rev of a functional spec.

          Touches the following files:

          M java/engine/org/apache/derby/vti/VTITemplate.java
          M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-13-aa-indexViews.diff. This patch adds support for creating Lucene indexes on arbitrary views. In order to index a view, you must supply a list of columns which will be treated as a key. I could not think of any reason to avoid sharing this capability with tables too. So now you can index a table which doesn't have a primary key. If you omit the key columns, then we try to find a primary key; and we raise an error if we can't find one. You do not supply key columns when you update a Lucene index, however. Instead, the plugin looks up the existing key information for the table function. Errors can, of course, occur if you drop a key column from the table. If you do this, you will get an error when you try to select from the Lucene index; that is because we still check to see if you have SELECT privilege on the missing column. This raises an interesting defect of the plugin: Creating a Lucene index does not prevent you from performing DDL on the table/view which will make it impossible to join the Lucene index back to Derby data. You are on your own there. At this point, I think that the api for the plugin is stable enough that I can write a first rev of a functional spec. Touches the following files: M java/engine/org/apache/derby/vti/VTITemplate.java M java/optional/org/apache/derby/optional/lucene/LuceneSupport.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          Hide
          ASF subversion and git services added a comment -

          Commit 1583505 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1583505 ]

          DERBY-590: Move the Lucene plugin into a new derbyoptionaltools.jar file; commit derby-590-12-aa-newJar.diff.

          Show
          ASF subversion and git services added a comment - Commit 1583505 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1583505 ] DERBY-590 : Move the Lucene plugin into a new derbyoptionaltools.jar file; commit derby-590-12-aa-newJar.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-12-aa-newJar.diff. This patch moves the Lucene plugin into a new jar file for optional tools.

          This patch introduces a new jar file: derbyoptionaltools.jar. Right now, this just holds the Lucene plugin, but we may be able to move other optional tools into this jar file too.

          I have not tested the release machinery, but a cursory glance at the release targets suggests that the new jar file will be automatically bundled into the derby distributions.

          After applying this patch, you will want to make sure that your testing classpath includes the new jar file.

          Touches the following files:

          ----------------

          M build.xml
          M tools/jar/extraDBMSclasses.properties

          Move the Lucene plugin classes from derby.jar into derbyoptionaltools.jar.

          ----------------

          M java/tools/org/apache/derby/tools/sysinfo.java
          M java/tools/org/apache/derby/impl/tools/sysinfo/Main.java
          M java/engine/org/apache/derby/iapi/services/info/ProductGenusNames.java

          Make sysinfo print out version information for derbyoptionaltools.jar.

          ----------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy

          Add a new grant block for derbyoptionaltools.jar to the policy file for LuceneSupportPermsTest. Move the lucene privileges from derby.jar to derbyoptionaltools.jar.

          Show
          Rick Hillegas added a comment - Attaching derby-590-12-aa-newJar.diff. This patch moves the Lucene plugin into a new jar file for optional tools. This patch introduces a new jar file: derbyoptionaltools.jar. Right now, this just holds the Lucene plugin, but we may be able to move other optional tools into this jar file too. I have not tested the release machinery, but a cursory glance at the release targets suggests that the new jar file will be automatically bundled into the derby distributions. After applying this patch, you will want to make sure that your testing classpath includes the new jar file. Touches the following files: ---------------- M build.xml M tools/jar/extraDBMSclasses.properties Move the Lucene plugin classes from derby.jar into derbyoptionaltools.jar. ---------------- M java/tools/org/apache/derby/tools/sysinfo.java M java/tools/org/apache/derby/impl/tools/sysinfo/Main.java M java/engine/org/apache/derby/iapi/services/info/ProductGenusNames.java Make sysinfo print out version information for derbyoptionaltools.jar. ---------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy Add a new grant block for derbyoptionaltools.jar to the policy file for LuceneSupportPermsTest. Move the lucene privileges from derby.jar to derbyoptionaltools.jar.
          Hide
          ASF subversion and git services added a comment -

          Commit 1583216 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1583216 ]

          DERBY-590: Move Lucene code to a new directory tree intended for optional code; commit derby-590-11-aa-moveCode.diff.

          Show
          ASF subversion and git services added a comment - Commit 1583216 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1583216 ] DERBY-590 : Move Lucene code to a new directory tree intended for optional code; commit derby-590-11-aa-moveCode.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-11-aa-moveCode.diff. This patch moves the Lucene plugin to the directory structure which Mike and I agreed on.

          Touches the following files:

          D java/engine/org/apache/derby/optional
          D java/engine/org/apache/derby/optional/LuceneUtils.java
          D java/engine/org/apache/derby/optional/build.xml
          M java/engine/org/apache/derby/impl/sql/compile/StaticMethodCallNode.java
          D java/engine/org/apache/derby/impl/optional/lucene
          D java/engine/org/apache/derby/impl/optional/lucene/LuceneQueryVTI.java
          D java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java
          D java/engine/org/apache/derby/impl/optional/lucene/LuceneListIndexesVTI.java
          D java/engine/org/apache/derby/impl/optional/lucene/build.xml
          M java/engine/org/apache/derby/impl/build.xml
          M java/engine/org/apache/derby/catalog/Java5SystemProcedures.java
          M java/engine/build.xml
          A java/optional
          A java/optional/org
          A java/optional/org/apache
          A java/optional/org/apache/derby
          A java/optional/org/apache/derby/optional
          A java/optional/org/apache/derby/optional/api
          A java/optional/org/apache/derby/optional/api/LuceneUtils.java
          A java/optional/org/apache/derby/optional/lucene
          A java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java
          A java/optional/org/apache/derby/optional/lucene/LuceneSupport.java
          A java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java
          A java/optional/build.xml
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M build.xml
          M tools/ant/properties/dirs.properties
          M tools/jar/extraDBMSclasses.properties

          Show
          Rick Hillegas added a comment - Attaching derby-590-11-aa-moveCode.diff. This patch moves the Lucene plugin to the directory structure which Mike and I agreed on. Touches the following files: D java/engine/org/apache/derby/optional D java/engine/org/apache/derby/optional/LuceneUtils.java D java/engine/org/apache/derby/optional/build.xml M java/engine/org/apache/derby/impl/sql/compile/StaticMethodCallNode.java D java/engine/org/apache/derby/impl/optional/lucene D java/engine/org/apache/derby/impl/optional/lucene/LuceneQueryVTI.java D java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java D java/engine/org/apache/derby/impl/optional/lucene/LuceneListIndexesVTI.java D java/engine/org/apache/derby/impl/optional/lucene/build.xml M java/engine/org/apache/derby/impl/build.xml M java/engine/org/apache/derby/catalog/Java5SystemProcedures.java M java/engine/build.xml A java/optional A java/optional/org A java/optional/org/apache A java/optional/org/apache/derby A java/optional/org/apache/derby/optional A java/optional/org/apache/derby/optional/api A java/optional/org/apache/derby/optional/api/LuceneUtils.java A java/optional/org/apache/derby/optional/lucene A java/optional/org/apache/derby/optional/lucene/LuceneQueryVTI.java A java/optional/org/apache/derby/optional/lucene/LuceneSupport.java A java/optional/org/apache/derby/optional/lucene/LuceneListIndexesVTI.java A java/optional/build.xml M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M build.xml M tools/ant/properties/dirs.properties M tools/jar/extraDBMSclasses.properties
          Hide
          ASF subversion and git services added a comment -

          Commit 1583200 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1583200 ]

          DERBY-590: Fix a locale-sensitive Lucene test; commit derby-590-10-aa-fixLocaleTest.diff.

          Show
          ASF subversion and git services added a comment - Commit 1583200 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1583200 ] DERBY-590 : Fix a locale-sensitive Lucene test; commit derby-590-10-aa-fixLocaleTest.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-10-aa-fixLocaleTest.diff. Hopefully this patch fixes a failure seen in the nightly tests after committing the previous patch (derby-590-09-aa-localeSensitiveAnalysis.diff).

          It appears that Locale "en_US" is not available on all of our test machines. The database is created with Locale "en" instead. That won't affect the behavior of the plugin because the analyzer it picks depends on the database language, not its country. The patch changes the assertion to just check for the leading two characters of the Locale string.

          There were other errors in the Lucene tests in the nightly runs on Windows. I don't understand those errors yet. I'm going to hold off filing a bug for those errors until after I fix this error which I think I understand.

          Touches the following file:

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-10-aa-fixLocaleTest.diff. Hopefully this patch fixes a failure seen in the nightly tests after committing the previous patch (derby-590-09-aa-localeSensitiveAnalysis.diff). It appears that Locale "en_US" is not available on all of our test machines. The database is created with Locale "en" instead. That won't affect the behavior of the plugin because the analyzer it picks depends on the database language, not its country. The patch changes the assertion to just check for the leading two characters of the Locale string. There were other errors in the Lucene tests in the nightly runs on Windows. I don't understand those errors yet. I'm going to hold off filing a bug for those errors until after I fix this error which I think I understand. Touches the following file: M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          Hide
          ASF subversion and git services added a comment -

          Commit 1583032 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1583032 ]

          DERBY-590: Make the Lucene plugin locale-sensitive; commit derby-590-09-aa-localeSensitiveAnalysis.diff.

          Show
          ASF subversion and git services added a comment - Commit 1583032 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1583032 ] DERBY-590 : Make the Lucene plugin locale-sensitive; commit derby-590-09-aa-localeSensitiveAnalysis.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-09-aa-localeSensitiveAnalysis.diff. This patch adds support for locale-sensitive searching.

          I was in the middle of this patch when Mike and I agreed on a new directory scheme. This patch uses the old directory scheme. In a later patch, I will move the code into the new scheme.

          Before describing the patch in greater detail, let me first describe the Lucene Analyzer interface. The Analyzer is the object which turns a block of text into a list of indexable terms. The Analyzer is used when you initially index text. The Analyzer is also used later on when you run a query. It is expected that you will use the same Analyzer to parse the text when you index it and when you query it. Lucene supplies a number of locale-sensitive Analyzers. They are listed in the LuceneUtils class included in this patch.

          This patch adds a new argument to the createIndex() and updateIndex() procedures. The new argument is a (possibly null) method name. It is the name of a static, public, no-arg method which instantiates a Lucene Analyzer. If you leave this argument null, then it defaults to org.apache.derby.optional.LuceneUtils.defaultAnalyzer(). That method looks for a Lucene Analyzer which matches the database Locale. If a good match can't be found, then the method returns StandardAnalyzer, the Analyzer which the plugin has been using up until now.

          The tests return slightly different (and I think better) results now that they use a locale-sensitive Analyzer.

          At this point, the Lucene index needs to be more stateful. In particular, the index needs to know what Analyzer created it so that the same Analyzer can be re-used at query time. So this patch introduces a derby-lucene.properties file. This file is added to the index directory by createIndex() and it is re-written by updateIndex(). Right now, the properties file contains the following information:

          o The version number of the Lucene software which created the index.

          o The timestamp when the index was created/updated.

          o The name of the static method which instantiated the Analyzer.

          o The name of the Analyzer which was instantiated.

          This extra information is added to the ResultSet returned by the listIndexes() table function.

          Touches the following files:

          ------------

          A java/engine/org/apache/derby/optional
          A java/engine/org/apache/derby/optional/LuceneUtils.java
          A java/engine/org/apache/derby/optional/build.xml
          M java/engine/build.xml

          New utility class, to be part of the public api eventually. Right now, this class provides support for looking up locale-sensitive Analyzers. We may think of other utility methods to add later on. I have not exposed this class in the public api yet. That will happen at the end when the plugin is ready to be documented and exposed to users.

          ------------

          M java/engine/org/apache/derby/impl/optional/lucene/LuceneQueryVTI.java
          M java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java
          M java/engine/org/apache/derby/impl/optional/lucene/LuceneListIndexesVTI.java
          M java/engine/org/apache/derby/impl/optional/lucene/build.xml

          The argument signatures of createIndex() and updateIndex() have changed. The signature of the table returned by listIndexes() has changed too.

          ------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java
          M build.xml

          The tests reflect the new signatures. New tests have been added for the new funtionality. The decorator for LuceneSupportPermsTest now explicitly sets the Locale to en_US. Hopefully the decorators will compose correctly when run on machines having different default locales.

          Show
          Rick Hillegas added a comment - Attaching derby-590-09-aa-localeSensitiveAnalysis.diff. This patch adds support for locale-sensitive searching. I was in the middle of this patch when Mike and I agreed on a new directory scheme. This patch uses the old directory scheme. In a later patch, I will move the code into the new scheme. Before describing the patch in greater detail, let me first describe the Lucene Analyzer interface. The Analyzer is the object which turns a block of text into a list of indexable terms. The Analyzer is used when you initially index text. The Analyzer is also used later on when you run a query. It is expected that you will use the same Analyzer to parse the text when you index it and when you query it. Lucene supplies a number of locale-sensitive Analyzers. They are listed in the LuceneUtils class included in this patch. This patch adds a new argument to the createIndex() and updateIndex() procedures. The new argument is a (possibly null) method name. It is the name of a static, public, no-arg method which instantiates a Lucene Analyzer. If you leave this argument null, then it defaults to org.apache.derby.optional.LuceneUtils.defaultAnalyzer(). That method looks for a Lucene Analyzer which matches the database Locale. If a good match can't be found, then the method returns StandardAnalyzer, the Analyzer which the plugin has been using up until now. The tests return slightly different (and I think better) results now that they use a locale-sensitive Analyzer. At this point, the Lucene index needs to be more stateful. In particular, the index needs to know what Analyzer created it so that the same Analyzer can be re-used at query time. So this patch introduces a derby-lucene.properties file. This file is added to the index directory by createIndex() and it is re-written by updateIndex(). Right now, the properties file contains the following information: o The version number of the Lucene software which created the index. o The timestamp when the index was created/updated. o The name of the static method which instantiated the Analyzer. o The name of the Analyzer which was instantiated. This extra information is added to the ResultSet returned by the listIndexes() table function. Touches the following files: ------------ A java/engine/org/apache/derby/optional A java/engine/org/apache/derby/optional/LuceneUtils.java A java/engine/org/apache/derby/optional/build.xml M java/engine/build.xml New utility class, to be part of the public api eventually. Right now, this class provides support for looking up locale-sensitive Analyzers. We may think of other utility methods to add later on. I have not exposed this class in the public api yet. That will happen at the end when the plugin is ready to be documented and exposed to users. ------------ M java/engine/org/apache/derby/impl/optional/lucene/LuceneQueryVTI.java M java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java M java/engine/org/apache/derby/impl/optional/lucene/LuceneListIndexesVTI.java M java/engine/org/apache/derby/impl/optional/lucene/build.xml The argument signatures of createIndex() and updateIndex() have changed. The signature of the table returned by listIndexes() has changed too. ------------ M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java M build.xml The tests reflect the new signatures. New tests have been added for the new funtionality. The decorator for LuceneSupportPermsTest now explicitly sets the Locale to en_US. Hopefully the decorators will compose correctly when run on machines having different default locales.
          Hide
          ASF subversion and git services added a comment -

          Commit 1582514 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1582514 ]

          DERBY-590: Add flag for running tests without the Lucene jar files; commit derby-590-08-aa-omitLuceneFlag.diff.

          Show
          ASF subversion and git services added a comment - Commit 1582514 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1582514 ] DERBY-590 : Add flag for running tests without the Lucene jar files; commit derby-590-08-aa-omitLuceneFlag.diff.
          Hide
          Rick Hillegas added a comment -

          Default and Omitted test runs passed cleanly for me on derby-590-08-aa-omitLuceneFlag.diff.

          Show
          Rick Hillegas added a comment - Default and Omitted test runs passed cleanly for me on derby-590-08-aa-omitLuceneFlag.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-08-aa-omitLuceneFlag.diff. This patch adds a flag to cause the tests to omit the Lucene plugin tests and at the same time demand that the Lucene jars not be present on the classpath.

          This patch introduces a new Derby testing flag: derby.tests.omitLucene. When this flag is set to true, the Lucene tests are NOT run and the LuceneSuite asserts that the Lucene jar files must NOT be on the classpath.

          I have run the Lucene suite in the following configurations with the following (expected) results:

          o Default - The flag is not set and the Lucene jars ARE on the classpath. The LuceneSuite runs cleanly.

          o Missing - The flag is not set and the Lucene jars are NOT on the classpath. The LuceneSuite raises many ClassNotFoundExceptions.

          o Present - The flag is set and the Lucene jars ARE on the classpath. The LuceneSuite raises an error, complaining that it can see the jar files.

          o Omitted - The flag is set and the Lucene jars are NOT on the classpath. The LuceneSuite runs cleanly but with 0 tests.

          I will run the full test suite in the Default and Omitted configurations.

          Touches the following files:

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java
          M java/testing/org/apache/derbyTesting/junit/JDBC.java
          M java/testing/org/apache/derbyTesting/junit/TestConfiguration.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-08-aa-omitLuceneFlag.diff. This patch adds a flag to cause the tests to omit the Lucene plugin tests and at the same time demand that the Lucene jars not be present on the classpath. This patch introduces a new Derby testing flag: derby.tests.omitLucene. When this flag is set to true, the Lucene tests are NOT run and the LuceneSuite asserts that the Lucene jar files must NOT be on the classpath. I have run the Lucene suite in the following configurations with the following (expected) results: o Default - The flag is not set and the Lucene jars ARE on the classpath. The LuceneSuite runs cleanly. o Missing - The flag is not set and the Lucene jars are NOT on the classpath. The LuceneSuite raises many ClassNotFoundExceptions. o Present - The flag is set and the Lucene jars ARE on the classpath. The LuceneSuite raises an error, complaining that it can see the jar files. o Omitted - The flag is set and the Lucene jars are NOT on the classpath. The LuceneSuite runs cleanly but with 0 tests. I will run the full test suite in the Default and Omitted configurations. Touches the following files: M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java M java/testing/org/apache/derbyTesting/junit/JDBC.java M java/testing/org/apache/derbyTesting/junit/TestConfiguration.java
          Hide
          Rick Hillegas added a comment - - edited

          Hi Mike,

          The modular separation you want is largely observed by the Lucene plugin right now. The exceptions are:

          o org.apache.derby.iapi.sql.dictionary.OptionalTool is referenced because the plugin implements this interface. Now that we have agreement on a source code tree for optional tools, this interface could probably be moved over there and perhaps eventually exposed in the public api.

          o org.apache.derby.iapi.util.IdUtil is referenced because the plugin needs the identifier normalization logic from IdUtil in order to convert string arguments into SQL identifiers and avoid SQL injection attacks.

          o org.apache.derby.iapi.error.StandardException and org.apache.derby.shared.common.reference.SQLState are referenced so that the plugin can construct localized SQLExceptions with SQLStates.

          o org.apache.derby.iapi.sql.conn.ConnectionUtil and org.apache.derby.iapi.sql.conn.LanguageConnectionContext are referenced in order to retrieve the database Locale.

          o org.apache.derby.impl.jdbc.EmbedConnection is referenced in order to retrieve the location of the database directory. This is the only instance of the plugin reaching into the impl classes. I will see whether this can be improved.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - - edited Hi Mike, The modular separation you want is largely observed by the Lucene plugin right now. The exceptions are: o org.apache.derby.iapi.sql.dictionary.OptionalTool is referenced because the plugin implements this interface. Now that we have agreement on a source code tree for optional tools, this interface could probably be moved over there and perhaps eventually exposed in the public api. o org.apache.derby.iapi.util.IdUtil is referenced because the plugin needs the identifier normalization logic from IdUtil in order to convert string arguments into SQL identifiers and avoid SQL injection attacks. o org.apache.derby.iapi.error.StandardException and org.apache.derby.shared.common.reference.SQLState are referenced so that the plugin can construct localized SQLExceptions with SQLStates. o org.apache.derby.iapi.sql.conn.ConnectionUtil and org.apache.derby.iapi.sql.conn.LanguageConnectionContext are referenced in order to retrieve the database Locale. o org.apache.derby.impl.jdbc.EmbedConnection is referenced in order to retrieve the location of the database directory. This is the only instance of the plugin reaching into the impl classes. I will see whether this can be improved. Thanks, -Rick
          Hide
          Mike Matrigali added a comment -

          the proposed locations seems clean to me. It would be good to insure that this code is modular and does not rely on any other derby imports if possible (especially non protocol imports), just public documented interfaces that any user program could do.

          Show
          Mike Matrigali added a comment - the proposed locations seems clean to me. It would be good to insure that this code is modular and does not rely on any other derby imports if possible (especially non protocol imports), just public documented interfaces that any user program could do.
          Hide
          ASF subversion and git services added a comment -

          Commit 1582178 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1582178 ]

          DERBY-590: Grant another privilege to Lucene; commit derby-590-07-aa-accessClassInPackage.sun.misc.diff.

          Show
          ASF subversion and git services added a comment - Commit 1582178 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1582178 ] DERBY-590 : Grant another privilege to Lucene; commit derby-590-07-aa-accessClassInPackage.sun.misc.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-08-aa-localeSensitiveAnalysis.diff. This grants another privilege to Lucene and the LuceneSupportPermsTest:

          permission java.lang.RuntimePermission "accessClassInPackage.sun.misc";

          Touches the following file:

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy

          Show
          Rick Hillegas added a comment - Attaching derby-590-08-aa-localeSensitiveAnalysis.diff. This grants another privilege to Lucene and the LuceneSupportPermsTest: permission java.lang.RuntimePermission "accessClassInPackage.sun.misc"; Touches the following file: M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy
          Hide
          Rick Hillegas added a comment -

          Here is a proposal for where to include the optional code in the source tree:

          java/
            build/
              ...
            client/
              ...
            demo/
              ...
            drda/
              ...
            engine/
              ...
            optional/
              org/
                apache/
                  derby/
                    optional/
                      api/
                      lucene/
                  
            shared/
              ...
            storeless/
              ...
            stubs/
              ...
            testing/
              ...
            tools/
              ...
          

          The derbyOptionalTools.jar file would hold the classes under java/optional/org/apache/derby/optional. Select classes in org.apache.derby.optional.api would go into the public api.

          What are your thoughts?

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Here is a proposal for where to include the optional code in the source tree: java/ build/ ... client/ ... demo/ ... drda/ ... engine/ ... optional/ org/ apache/ derby/ optional/ api/ lucene/ shared/ ... storeless/ ... stubs/ ... testing/ ... tools/ ... The derbyOptionalTools.jar file would hold the classes under java/optional/org/apache/derby/optional. Select classes in org.apache.derby.optional.api would go into the public api. What are your thoughts? Thanks, -Rick
          Hide
          Rick Hillegas added a comment -

          Thanks, Mike. If you are comfortable with my moving the plugin to a new derbyOptionalTools.jar file, then I will do that. Over time I may migrate some other optional code there. Thanks.

          Show
          Rick Hillegas added a comment - Thanks, Mike. If you are comfortable with my moving the plugin to a new derbyOptionalTools.jar file, then I will do that. Over time I may migrate some other optional code there. Thanks.
          Hide
          ASF subversion and git services added a comment -

          Commit 1582090 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1582090 ]

          DERBY-590: Add another privilege to the test policy for the Lucene plugin; commit derby-590-06-aa-suppressAccessChecks.diff.

          Show
          ASF subversion and git services added a comment - Commit 1582090 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1582090 ] DERBY-590 : Add another privilege to the test policy for the Lucene plugin; commit derby-590-06-aa-suppressAccessChecks.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-06-aa-suppressAccessChecks.diff. Next round of privilege whackamole for the Lucene plugin.

          The following permission is granted to the app (the LuceneSupportPermsTest) and to the Lucene core jar file:

          permission java.lang.reflect.ReflectPermission "suppressAccessChecks";

          Touches the following file:

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy

          Show
          Rick Hillegas added a comment - Attaching derby-590-06-aa-suppressAccessChecks.diff. Next round of privilege whackamole for the Lucene plugin. The following permission is granted to the app (the LuceneSupportPermsTest) and to the Lucene core jar file: permission java.lang.reflect.ReflectPermission "suppressAccessChecks"; Touches the following file: M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy
          Hide
          Mike Matrigali added a comment -

          I continue to think that derby.jar is the wrong place for this tool and
          other optional tools. I do not feel strongly it need go into derbytools.jar,
          just that it should go in some other jar. Functionally I am looking to ways
          to insure new optional tools do not add to code bloat (both disk and runtime)
          to base server users who do not want the optional tools. Looking to insure
          this by having separate jars and by being able to run full sets of non-optional
          tools tests with a classpath that does not include these optional tool jars.
          >
          > Or maybe I misunderstand you. Are you suggesting that we crack open the Lucene
          jars and put all their contents together with the plugin into a single Lucene-s
          pecific jar? I think there will be problems with that approach.
          No, I am not suggesting this - we should leave lucene jars alone. I assume
          this project does not need any lucene jar changes, just uses existing
          interfaces. I am hoping similarly the lucene optional tool need not have
          derby changes, just use existing generic optional tool interfaces (which may
          need to be altered as we learn new interfaces to provide for all new tools).
          >
          > Are you suggesting that we allocate a separate jar file to each optional tool?
          I did suggest this as a point for discussion. I don't have strong
          opinions one way or another as long as the libraries and dependencies are
          separate from the base server. Some optional tools may make more sense
          to group together, like the various existing diagnostic tools. To me lucerne
          seems very different.
          >
          > Yes, please, I would like more discussion about the transactional behavior of
          the Lucene plugin. Having another set of eyes on this would be very helpful. In
          particular, I want to reduce the chance that users will experience deadlocks bet
          ween Lucene's locks and Derby's locks.
          I have to admit at this point I have not looked very closely at this other
          than to note that the implementation seems not finished, and seemed to allow
          for the index to not be up to date with the data (but would get there
          eventually). I am fine with incremental development in the trunk.

          The index not being in sync with the data is not going to be SQL standard so
          was hoping it would not be part of the main product.
          I do see that lucene searching in a non-SQL
          compatible way may be quite useful to some users, so did not want to get
          in the way - but still see it as something different than the derby project.

          Show
          Mike Matrigali added a comment - I continue to think that derby.jar is the wrong place for this tool and other optional tools. I do not feel strongly it need go into derbytools.jar, just that it should go in some other jar. Functionally I am looking to ways to insure new optional tools do not add to code bloat (both disk and runtime) to base server users who do not want the optional tools. Looking to insure this by having separate jars and by being able to run full sets of non-optional tools tests with a classpath that does not include these optional tool jars. > > Or maybe I misunderstand you. Are you suggesting that we crack open the Lucene jars and put all their contents together with the plugin into a single Lucene-s pecific jar? I think there will be problems with that approach. No, I am not suggesting this - we should leave lucene jars alone. I assume this project does not need any lucene jar changes, just uses existing interfaces. I am hoping similarly the lucene optional tool need not have derby changes, just use existing generic optional tool interfaces (which may need to be altered as we learn new interfaces to provide for all new tools). > > Are you suggesting that we allocate a separate jar file to each optional tool? I did suggest this as a point for discussion. I don't have strong opinions one way or another as long as the libraries and dependencies are separate from the base server. Some optional tools may make more sense to group together, like the various existing diagnostic tools. To me lucerne seems very different. > > Yes, please, I would like more discussion about the transactional behavior of the Lucene plugin. Having another set of eyes on this would be very helpful. In particular, I want to reduce the chance that users will experience deadlocks bet ween Lucene's locks and Derby's locks. I have to admit at this point I have not looked very closely at this other than to note that the implementation seems not finished, and seemed to allow for the index to not be up to date with the data (but would get there eventually). I am fine with incremental development in the trunk. The index not being in sync with the data is not going to be SQL standard so was hoping it would not be part of the main product. I do see that lucene searching in a non-SQL compatible way may be quite useful to some users, so did not want to get in the way - but still see it as something different than the derby project.
          Hide
          ASF subversion and git services added a comment -

          Commit 1581935 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1581935 ]

          DERBY-590: Fix two problems seen in LuceneSupportPermsTest during the continuous integregation and nightly tests; commit derby-590-05-aa-accessDeclaredMembers.diff.

          Show
          ASF subversion and git services added a comment - Commit 1581935 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1581935 ] DERBY-590 : Fix two problems seen in LuceneSupportPermsTest during the continuous integregation and nightly tests; commit derby-590-05-aa-accessDeclaredMembers.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-05-aa-accessDeclaredMembers.diff. This patch attempts to fix two problems observed in the platform tests:

          1) On certain operating systems (including Linux variants), the following permission must be granted both to the application (the test) and to the Lucene core jar in order to get LuceneSupportPermsTest to run cleanly:

          permission java.lang.RuntimePermission "accessDeclaredMembers";

          2) On Java 7 we are seeing errors when the test cases in LuceneSupportPermsTest run in an unexpected order. The errors arise because a user schema does not exist yet. The fix is to create a table in that schema in order to force schema creation. A longer-term fix will be to figure out why Derby is demanding that a user schema exist. But this should fix the test for the moment.

          Touches the following files:

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-05-aa-accessDeclaredMembers.diff. This patch attempts to fix two problems observed in the platform tests: 1) On certain operating systems (including Linux variants), the following permission must be granted both to the application (the test) and to the Lucene core jar in order to get LuceneSupportPermsTest to run cleanly: permission java.lang.RuntimePermission "accessDeclaredMembers"; 2) On Java 7 we are seeing errors when the test cases in LuceneSupportPermsTest run in an unexpected order. The errors arise because a user schema does not exist yet. The fix is to create a table in that schema in order to force schema creation. A longer-term fix will be to figure out why Derby is demanding that a user schema exist. But this should fix the test for the moment. Touches the following files: M java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          Hide
          Rick Hillegas added a comment -

          I would support moving the Lucene plugin (and maybe other optional tools) into a new derbyOptionalTools.jar. I continue to think that derbytools.jar is the wrong place for this tool and some of the other optional tools. That is because some of the optional tools (including the Lucene plugin) rely on engine classes. derbytools.jar is supposed to hold tools which run fine on a client machine without the engine jar.

          Or maybe I misunderstand you. Are you suggesting that we crack open the Lucene jars and put all their contents together with the plugin into a single Lucene-specific jar? I think there will be problems with that approach.

          Are you suggesting that we allocate a separate jar file to each optional tool?

          Yes, please, I would like more discussion about the transactional behavior of the Lucene plugin. Having another set of eyes on this would be very helpful. In particular, I want to reduce the chance that users will experience deadlocks between Lucene's locks and Derby's locks.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - I would support moving the Lucene plugin (and maybe other optional tools) into a new derbyOptionalTools.jar. I continue to think that derbytools.jar is the wrong place for this tool and some of the other optional tools. That is because some of the optional tools (including the Lucene plugin) rely on engine classes. derbytools.jar is supposed to hold tools which run fine on a client machine without the engine jar. Or maybe I misunderstand you. Are you suggesting that we crack open the Lucene jars and put all their contents together with the plugin into a single Lucene-specific jar? I think there will be problems with that approach. Are you suggesting that we allocate a separate jar file to each optional tool? Yes, please, I would like more discussion about the transactional behavior of the Lucene plugin. Having another set of eyes on this would be very helpful. In particular, I want to reduce the chance that users will experience deadlocks between Lucene's locks and Derby's locks. Thanks, -Rick
          Hide
          Mike Matrigali added a comment -

          I see that the optional tool for lucene is being added to derby.jar. I continue to believe that optional tools
          should be optional to the jar classpath as well as the runtime loading of the optional tool. Support specific
          to lucene should to into a separate jar, either an derbyoptionallucene.jar (or whatever would be a good name) or I would be ok for now to lump it into derbytools.jar.

          I would like to see a sustainable model where developers can make an unbounded number of optional
          tools avaliable for derby, but each one should not add to the weight of the core derby.jar.

          Also I would prefer that these tools be in a different project than the derby server and the interfaces in
          derby be enhanced so that is possible.

          While I think the code being donated for the lucene integration looks interesting, I think we need to be
          clear about its functionality. I have not reviewed the code as I believed it was going to exist as an OPTIONAL tool, not integrated into the product. Some high level descriptions indicate that its indexes
          are not transactional, which does not mesh well with default current derby behavior.

          I continue to hope that we can come up with a project or separate "demo" set of jars for features like this
          so that users can benefit from a shared development environment. But also so that we can continue
          to promote the stable core of the derby server with clear and clean interfaces to allow the less supported
          functions to work (user provided new functions, user provided optional tools, ...)

          Show
          Mike Matrigali added a comment - I see that the optional tool for lucene is being added to derby.jar. I continue to believe that optional tools should be optional to the jar classpath as well as the runtime loading of the optional tool. Support specific to lucene should to into a separate jar, either an derbyoptionallucene.jar (or whatever would be a good name) or I would be ok for now to lump it into derbytools.jar. I would like to see a sustainable model where developers can make an unbounded number of optional tools avaliable for derby, but each one should not add to the weight of the core derby.jar. Also I would prefer that these tools be in a different project than the derby server and the interfaces in derby be enhanced so that is possible. While I think the code being donated for the lucene integration looks interesting, I think we need to be clear about its functionality. I have not reviewed the code as I believed it was going to exist as an OPTIONAL tool, not integrated into the product. Some high level descriptions indicate that its indexes are not transactional, which does not mesh well with default current derby behavior. I continue to hope that we can come up with a project or separate "demo" set of jars for features like this so that users can benefit from a shared development environment. But also so that we can continue to promote the stable core of the derby server with clear and clean interfaces to allow the less supported functions to work (user provided new functions, user provided optional tools, ...)
          Hide
          ASF subversion and git services added a comment -

          Commit 1581312 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1581312 ]

          DERBY-590: Remove the ID column from the Lucene plugin listIndexes() table function; commit derby-590-04-aa-removeIDFromListIndexes.diff.

          Show
          ASF subversion and git services added a comment - Commit 1581312 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1581312 ] DERBY-590 : Remove the ID column from the Lucene plugin listIndexes() table function; commit derby-590-04-aa-removeIDFromListIndexes.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-04-aa-removeIDFromListIndexes.diff. This patch removes the ID column from the Lucene plugin's listIndexes() table function. Hopefully, this will fix the non-deterministic results we're seeing in the platform tests.

          The listIndexes() table function makes up an index id for each Lucene index. However, the id assigned to a particular index really has no durable meaning and is completely arbitrary. I have removed this column from the table function. I will give some thought to how or whether we should assign ids to the indexes.

          Touches the following files:

          --------------

          M java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java
          M java/engine/org/apache/derby/impl/optional/lucene/LuceneListIndexesVTI.java

          Remove the ID column.

          --------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java
          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java

          Adjust the tests.

          Show
          Rick Hillegas added a comment - Attaching derby-590-04-aa-removeIDFromListIndexes.diff. This patch removes the ID column from the Lucene plugin's listIndexes() table function. Hopefully, this will fix the non-deterministic results we're seeing in the platform tests. The listIndexes() table function makes up an index id for each Lucene index. However, the id assigned to a particular index really has no durable meaning and is completely arbitrary. I have removed this column from the table function. I will give some thought to how or whether we should assign ids to the indexes. Touches the following files: -------------- M java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java M java/engine/org/apache/derby/impl/optional/lucene/LuceneListIndexesVTI.java Remove the ID column. -------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java M java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java Adjust the tests.
          Hide
          ASF subversion and git services added a comment -

          Commit 1580611 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1580611 ]

          DERBY-590: Remove some debug cruft from the junit-single target; commit derby-590-03-aa-removeTestingDiagnostic.diff.

          Show
          ASF subversion and git services added a comment - Commit 1580611 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1580611 ] DERBY-590 : Remove some debug cruft from the junit-single target; commit derby-590-03-aa-removeTestingDiagnostic.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-03-aa-removeTestingDiagnostic.diff. This patch removes some debug cruft introduced by derby-590-01-am-publicAccessToLuceneRoutines.diff. The cruft was meant to verify that the junit-single ant target sets up a classpath which includes the Lucene jar files.

          Touches the following file:

          M build.xml

          Show
          Rick Hillegas added a comment - Attaching derby-590-03-aa-removeTestingDiagnostic.diff. This patch removes some debug cruft introduced by derby-590-01-am-publicAccessToLuceneRoutines.diff. The cruft was meant to verify that the junit-single ant target sets up a classpath which includes the Lucene jar files. Touches the following file: M build.xml
          Hide
          ASF subversion and git services added a comment -

          Commit 1580587 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1580587 ]

          DERBY-590: Cleanup some suspect coding practices disclosed by running the Findbugs lint tool; commit derby-590-02-aa-cleanupFindbugsErrors.diff.

          Show
          ASF subversion and git services added a comment - Commit 1580587 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1580587 ] DERBY-590 : Cleanup some suspect coding practices disclosed by running the Findbugs lint tool; commit derby-590-02-aa-cleanupFindbugsErrors.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-02-aa-cleanupFindbugsErrors.diff. This patch cleans up some suspect coding practices discovered by running the Findbugs lint tool.

          Touches the following files:

          M java/engine/org/apache/derby/impl/optional/lucene/LuceneQueryVTI.java
          M java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java

          Show
          Rick Hillegas added a comment - Attaching derby-590-02-aa-cleanupFindbugsErrors.diff. This patch cleans up some suspect coding practices discovered by running the Findbugs lint tool. Touches the following files: M java/engine/org/apache/derby/impl/optional/lucene/LuceneQueryVTI.java M java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java
          Hide
          ASF subversion and git services added a comment -

          Commit 1580387 from Rick Hillegas in branch 'code/trunk'
          [ https://svn.apache.org/r1580387 ]

          DERBY-590: First increment of support for Lucene indexing of Derby text columns; tests passed cleanly on derby-590-01-am-publicAccessToLuceneRoutines.diff.

          Show
          ASF subversion and git services added a comment - Commit 1580387 from Rick Hillegas in branch 'code/trunk' [ https://svn.apache.org/r1580387 ] DERBY-590 : First increment of support for Lucene indexing of Derby text columns; tests passed cleanly on derby-590-01-am-publicAccessToLuceneRoutines.diff.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-01-am-publicAccessToLuceneRoutines.diff. I am running tests against this version. Provided that the tests pass, I think that this version is ready for checkin as the first increment of Lucene support.

          This version adds the following:

          i) More tests while running under SQL authorization.

          ii) Support for running with collation turned on.

          iii) Better serialization of indexable datatypes, rather than representing everything as a string.

          iv) Cleaned-up javadoc.

          v) Various bug fixes.

          Of course, we are a long way from documenting this feature or exposing it to users. I hope to continue working on the following improvements:

          1) Add support for other languages. Right now the Lucene plugin assumes that the text is English.

          2) Make it possible to create Lucene indexes on views as well as base tables.

          3) Begin writing a functional spec, which will evolve as the plugin becomes more capable. The spec will describe what works and what doesn't.

          4) Restore Andrew's original feature, which allowed you to incrementally index new/updated documents via triggers.

          5) See if we can get the plugin to work with in-memory databases, backup, and encryption.

          Touches the following additional files:

          ---------

          A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java
          A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java

          Added a test for running the plugin with collation turned on. Put all the Lucene tests in a suite to facilitate testing them.

          Show
          Rick Hillegas added a comment - Attaching derby-590-01-am-publicAccessToLuceneRoutines.diff. I am running tests against this version. Provided that the tests pass, I think that this version is ready for checkin as the first increment of Lucene support. This version adds the following: i) More tests while running under SQL authorization. ii) Support for running with collation turned on. iii) Better serialization of indexable datatypes, rather than representing everything as a string. iv) Cleaned-up javadoc. v) Various bug fixes. Of course, we are a long way from documenting this feature or exposing it to users. I hope to continue working on the following improvements: 1) Add support for other languages. Right now the Lucene plugin assumes that the text is English. 2) Make it possible to create Lucene indexes on views as well as base tables. 3) Begin writing a functional spec, which will evolve as the plugin becomes more capable. The spec will describe what works and what doesn't. 4) Restore Andrew's original feature, which allowed you to incrementally index new/updated documents via triggers. 5) See if we can get the plugin to work with in-memory databases, backup, and encryption. Touches the following additional files: --------- A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSuite.java A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneCollationTest.java Added a test for running the plugin with collation turned on. Put all the Lucene tests in a suite to facilitate testing them.
          Hide
          Rick Hillegas added a comment -

          Attaching a revised version of the patch: derby-590-01-ah-publicAccessToLuceneRoutines.diff. This version wires in support for running the Lucene plugin under a Java SecurityManager. This version also adds the first set of tests for verifying the plugin's behavior when SQL Authorization is enabled and a SecurityManager is installed.

          I think that I have figured out a reasonably small set of permissions needed to run the plugin under a SecurityManager. The process of discovering this set is documented on https://issues.apache.org/jira/browse/LUCENE-5471.

          The following permissions must be granted to the Lucene core jar file...

          // Permissions for the Lucene plugin
          grant codeBase "${lucene.core.jar.url}"
          {
            // permissions for file access, write access only to sandbox:
            permission java.io.FilePermission "${databaseDirectory}${/}lucene", "read,write,delete";
            permission java.io.FilePermission "${databaseDirectory}${/}lucene${/}-", "read,write,delete";
            
            // Basic permissions needed for Lucene to work:
            permission java.util.PropertyPermission "user.dir", "read";
            permission java.util.PropertyPermission "sun.arch.data.model", "read";
          };
          

          ...and the following additional permissions must be granted to the Derby engine jar...

          grant codeBase "${derby.jar}"
          {
              ...
          
            // extra permissions needed for the Lucene plugin
            permission java.io.FilePermission "${lucene.core.jar.file}", "read";
            permission java.util.PropertyPermission "user.dir", "read";
          };
          

          For an example, see luceneSupport.policy.

          Touches the following additional files:

          A java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy
          A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java
          M java/testing/org/apache/derbyTesting/junit/SecurityManagerSetup.java

          Show
          Rick Hillegas added a comment - Attaching a revised version of the patch: derby-590-01-ah-publicAccessToLuceneRoutines.diff. This version wires in support for running the Lucene plugin under a Java SecurityManager. This version also adds the first set of tests for verifying the plugin's behavior when SQL Authorization is enabled and a SecurityManager is installed. I think that I have figured out a reasonably small set of permissions needed to run the plugin under a SecurityManager. The process of discovering this set is documented on https://issues.apache.org/jira/browse/LUCENE-5471 . The following permissions must be granted to the Lucene core jar file... // Permissions for the Lucene plugin grant codeBase "${lucene.core.jar.url}" { // permissions for file access, write access only to sandbox: permission java.io.FilePermission "${databaseDirectory}${/}lucene", "read,write,delete"; permission java.io.FilePermission "${databaseDirectory}${/}lucene${/}-", "read,write,delete"; // Basic permissions needed for Lucene to work: permission java.util.PropertyPermission "user.dir", "read"; permission java.util.PropertyPermission "sun.arch.data.model", "read"; }; ...and the following additional permissions must be granted to the Derby engine jar... grant codeBase "${derby.jar}" { ... // extra permissions needed for the Lucene plugin permission java.io.FilePermission "${lucene.core.jar.file}", "read"; permission java.util.PropertyPermission "user.dir", "read"; }; For an example, see luceneSupport.policy. Touches the following additional files: A java/testing/org/apache/derbyTesting/functionTests/tests/lang/luceneSupport.policy A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportPermsTest.java M java/testing/org/apache/derbyTesting/junit/SecurityManagerSetup.java
          Hide
          Rick Hillegas added a comment -

          Attaching derby-590-01-ag-publicAccessToLuceneRoutines.diff. This patch builds on Andrew's approach, wiring Lucene support into Derby's SQL authorization scheme. If this approach looks promising, I will write additional regression tests to verify the claims I make about this solution.

          Before discussing what this patch does, let me recap what it does NOT do:

          1) It does NOT encrypt Lucene indexes if the database is encrypted.

          2) It does NOT work with in-memory databases.

          3) It does NOT work with backup/restore.

          In addition, I have changed the meaning of the updateIndex() procedure and lost the very useful meaning which it used to have. There are at least two important application profiles which have different needs from an index-updating procedure:

          i) Update-intensive: These applications do not want to incur the performance hit of re-indexing a document every time it changes. These applications may work best with a bulk-reindexing cron job which executes when user activity is low. This is the use-case supported by the revised updateIndex() procedure: the Lucene index is dropped and completely recreated although schema objects connected to it are not bounced.

          ii) Read-intensive: These applications can incur the performance hit of re-indexing a document every time it changes. These applications are well served by the trigger-driven usage of updateIndex() provided in Andrew's original patch.

          I think that it may be possible to recover Andrew's original functionality. But this requires more familiarity with Lucene than I can claim. Maybe Andrew can propose some ideas and we can converge on a solution.

          So much for the caveats. Here are the changes made by the new patch:

          A) The layout of the Lucene support directory is changed. There is now a subdirectory for each schema which has indexed tables in it. Under that directory, there is a subdirectory for each table which has indexed columns. Under the table-specific directory, there is a subdirectory for each text column which is indexed. Those are the leaf directories which contain the actual Lucene files. So the directory structure looks like this:

          databaseName
              seg0
              log
              lucene
                  SCHEMA1
                      TABLE1
                          TEXT_COLUMN1
                          TEXT_COLUMN2
          ...
          

          B) The luceneQuery() table function has been changed to be context aware. See the work recently done on https://issues.apache.org/jira/browse/DERBY-6117. Users no longer invoke luceneQuery() directly. Instead, for each indexed column, we create a table function specific to that column. The table function has the name $schemaName.$tableName__$columnName. This is the crucial change which allows us to wire Lucene support up to Derby's SQL authorization. The schema owner can grant EXECUTE privilege on the column-specific table function. In addition, the underlying context-aware luceneQuery() machinery starts out by issuing a select against the text column and the key columns of the table. This ensures that users must enjoy SELECT privilege on all of those columns in order to run the column-specific table function.

          C) A text column can be indexed only if the table has a primary key. The primary key can have multiple columns in it and the columns can be any indexable datatype. The column-specific table function returns a data set which includes the whole primary key plus the Lucene document id plus the rank number (the relevance of that document according to Lucene's calculations). This makes it easy to join the table function to the original table in order to obtain more extensive results.

          D) The arguments to the Lucene support procedures are now case-insensitive SQL identifiers rather than case-sensitive strings. This is a departure from the convention followed by most Derby system procedures, but I think the departure is welcome and long overdue.

          E) The Lucene support has been moved out of the tools jar and into the engine jar. This fixes the big surprise I recorded on https://issues.apache.org/jira/browse/DERBY-6470. This change also made it possible to use IdUtil in order to implement the semantics described above in D).

          F) It is assumed that the Lucene jars will be checked into the Derby source tree and that the Lucene support classes will always be built. However, there is still no plan to include Lucene jars in Derby binary distributions. Users who want to enable the optional Lucene support will have to install the Lucene jars themselves.

          G) In general, the following convention has been followed for each supported operation: Transactional writes are performed before any calls are made to Lucene. This means that if the Lucene calls raise an error, then the transactional writes are rolled back. I think that following a consistent convention like this will make it easier for users to reason about how Lucene support behaves.

          H) Miscellaneous improvements have been made in the areas of code factoring and integration with the Java security manager.

          Here's how you use the revised tool:

          Let's say that you have a table with this shape...

          create table lucenetest.titles
          (
              ID int generated always as identity primary key,
              ISBN varchar(16),
              PRINTISBN varchar(16),
              title varchar(1024),
              subtitle varchar(1024),
              author varchar(1024),
              series varchar(1024),
              publisher varchar(1024),
              collections varchar(128),
              collections2 varchar(128)
          );
          

          The DBO loads the Lucene support tool as follows...

          call syscs_util.syscs_register_tool( 'luceneSupport', true );
          

          This creates the LuceneSupport schema, containing the following procedures and functions...

          create procedure LuceneSupport.createIndex
          (
              schemaname varchar( 128 ),
              tablename varchar( 128 ),
              textcolumn varchar( 128 )
          )
          parameter style java modifies sql data language java
          external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.createIndex';
          
          create procedure LuceneSupport.dropIndex
          (
              schemaname varchar( 128 ),
          	tablename varchar( 128 ),
          	textcolumn varchar( 128 )
          )
          parameter style java modifies sql data language java
          external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.dropIndex';
          
          create procedure LuceneSupport.updateIndex
          (
              schemaname varchar( 128 ),
          	tablename varchar( 128 ),
          	textcolumn varchar( 128 )
          )
          parameter style java reads sql data language java
          external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.updateIndex';
          
          create function LuceneSupport.listIndexes() returns table
          (
              id int,
          	schemaname char( 128 ),
          	tablename char( 128 ),
          	columnname char( 128 ),
          	lastupdated timestamp
          )
          language java parameter style DERBY_JDBC_RESULT_SET contains sql
          external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.listIndexes'
          

          The LUCENETEST user can then index a text column as follows. The DBO can do this too. However, no-one else can index the column. That is because the createIndex() procedure attempts to create a table function in the lucenetest schema. Other users do not enjoy that privilege...

          call LuceneSupport.createIndex( 'lucenetest', 'titles', 'title' );
          

          ...which creates the following directory structure...

          databaseName
              seg0
              log
              lucene
                  LUCENETEST
                      TITLES
                          TITLE
          ...
          

          ...and the following column-specific table function:

          create function lucenetest.titles__title( query varchar( 32672 ), rankCutoff double )
          returns table
          (
              ID int,
              documentID int,
          	rank double
          )
          language java parameter style derby_jdbc_result_set contains sql
          external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.luceneQuery';
          

          The LUCENETEST user can then query the Lucene index and join it to the original table as follows...

          select title, author, publisher, documentID, rank
          from lucenetest.titles t, table ( lucenetest.titles__title( 'grapes', 0 ) ) l
          where t.id = l.id;
          

          The LUCENETEST user (or the DBO but no-one else) can drop the Lucene index as follows...

          call LuceneSupport.dropIndex( 'lucenetest', 'titles','title' );
          

          ...which drops the lucene/LUCENETEST/TITLES/TITLE directory. Directory deletion cascades up. That is, if there are no more indexed columns in lucenetest.titles, then we also drop lucene/LUCENETEST/TITLES. And we drop lucene/LUCENETEST if there are no more indexed tables in the lucenetest schema.

          The DBO (and only the DBO) can unload Lucene support as follows...

          call syscs_util.syscs_register_tool( 'luceneSupport', false );
          

          ...which drops all procedures and table functions created by the tool. This command also drops the lucene subdirectory and everything under it.

          Touches the following files:

          --------------------

          M java/engine/org/apache/derby/impl/sql/compile/StaticMethodCallNode.java

          Adds the Lucene support package to the list of Derby packages whose entry points can be bound to user-defined routines.

          --------------------

          A java/engine/org/apache/derby/impl/optional
          A java/engine/org/apache/derby/impl/optional/lucene
          A java/engine/org/apache/derby/impl/optional/lucene/LuceneQueryVTI.java
          A java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java
          A java/engine/org/apache/derby/impl/optional/lucene/LuceneListIndexesVTI.java
          A java/engine/org/apache/derby/impl/optional/lucene/build.xml
          M java/engine/org/apache/derby/impl/build.xml
          M java/engine/org/apache/derby/catalog/Java5SystemProcedures.java

          The Lucene support optional tool.

          --------------------

          M java/engine/org/apache/derby/vti/VTITemplate.java

          Some additional support for context-aware table functions. This support allows the function to query the JDBC metadata for the shape of the ResultSet it returns.

          --------------------

          M java/engine/org/apache/derby/loc/messages.xml
          M java/shared/org/apache/derby/shared/common/reference/SQLState.java

          New error messages for Lucene support.

          --------------------

          M build.xml
          A tools/java/lucene-analyzers-common-4.5.0.jar
          A tools/java/lucene-queryparser-4.5.0.jar
          A tools/java/lucene-core-4.5.0.jar
          M tools/ant/properties/extrapath.properties
          M tools/jar/extraDBMSclasses.properties

          Miscellaneous build machinery.

          --------------------

          A tools/release/notices/lucene.txt

          The Lucene notice file which must be included in the Derby NOTICE file if we are to ship the lucene jars in Derby source distributions.

          --------------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/lang/_Suite.java
          A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java

          Initial tests.

          Show
          Rick Hillegas added a comment - Attaching derby-590-01-ag-publicAccessToLuceneRoutines.diff. This patch builds on Andrew's approach, wiring Lucene support into Derby's SQL authorization scheme. If this approach looks promising, I will write additional regression tests to verify the claims I make about this solution. Before discussing what this patch does, let me recap what it does NOT do: 1) It does NOT encrypt Lucene indexes if the database is encrypted. 2) It does NOT work with in-memory databases. 3) It does NOT work with backup/restore. In addition, I have changed the meaning of the updateIndex() procedure and lost the very useful meaning which it used to have. There are at least two important application profiles which have different needs from an index-updating procedure: i) Update-intensive: These applications do not want to incur the performance hit of re-indexing a document every time it changes. These applications may work best with a bulk-reindexing cron job which executes when user activity is low. This is the use-case supported by the revised updateIndex() procedure: the Lucene index is dropped and completely recreated although schema objects connected to it are not bounced. ii) Read-intensive: These applications can incur the performance hit of re-indexing a document every time it changes. These applications are well served by the trigger-driven usage of updateIndex() provided in Andrew's original patch. I think that it may be possible to recover Andrew's original functionality. But this requires more familiarity with Lucene than I can claim. Maybe Andrew can propose some ideas and we can converge on a solution. So much for the caveats. Here are the changes made by the new patch: A) The layout of the Lucene support directory is changed. There is now a subdirectory for each schema which has indexed tables in it. Under that directory, there is a subdirectory for each table which has indexed columns. Under the table-specific directory, there is a subdirectory for each text column which is indexed. Those are the leaf directories which contain the actual Lucene files. So the directory structure looks like this: databaseName seg0 log lucene SCHEMA1 TABLE1 TEXT_COLUMN1 TEXT_COLUMN2 ... B) The luceneQuery() table function has been changed to be context aware. See the work recently done on https://issues.apache.org/jira/browse/DERBY-6117 . Users no longer invoke luceneQuery() directly. Instead, for each indexed column, we create a table function specific to that column. The table function has the name $schemaName.$tableName__$columnName. This is the crucial change which allows us to wire Lucene support up to Derby's SQL authorization. The schema owner can grant EXECUTE privilege on the column-specific table function. In addition, the underlying context-aware luceneQuery() machinery starts out by issuing a select against the text column and the key columns of the table. This ensures that users must enjoy SELECT privilege on all of those columns in order to run the column-specific table function. C) A text column can be indexed only if the table has a primary key. The primary key can have multiple columns in it and the columns can be any indexable datatype. The column-specific table function returns a data set which includes the whole primary key plus the Lucene document id plus the rank number (the relevance of that document according to Lucene's calculations). This makes it easy to join the table function to the original table in order to obtain more extensive results. D) The arguments to the Lucene support procedures are now case-insensitive SQL identifiers rather than case-sensitive strings. This is a departure from the convention followed by most Derby system procedures, but I think the departure is welcome and long overdue. E) The Lucene support has been moved out of the tools jar and into the engine jar. This fixes the big surprise I recorded on https://issues.apache.org/jira/browse/DERBY-6470 . This change also made it possible to use IdUtil in order to implement the semantics described above in D). F) It is assumed that the Lucene jars will be checked into the Derby source tree and that the Lucene support classes will always be built. However, there is still no plan to include Lucene jars in Derby binary distributions. Users who want to enable the optional Lucene support will have to install the Lucene jars themselves. G) In general, the following convention has been followed for each supported operation: Transactional writes are performed before any calls are made to Lucene. This means that if the Lucene calls raise an error, then the transactional writes are rolled back. I think that following a consistent convention like this will make it easier for users to reason about how Lucene support behaves. H) Miscellaneous improvements have been made in the areas of code factoring and integration with the Java security manager. Here's how you use the revised tool: Let's say that you have a table with this shape... create table lucenetest.titles ( ID int generated always as identity primary key, ISBN varchar(16), PRINTISBN varchar(16), title varchar(1024), subtitle varchar(1024), author varchar(1024), series varchar(1024), publisher varchar(1024), collections varchar(128), collections2 varchar(128) ); The DBO loads the Lucene support tool as follows... call syscs_util.syscs_register_tool( 'luceneSupport', true ); This creates the LuceneSupport schema, containing the following procedures and functions... create procedure LuceneSupport.createIndex ( schemaname varchar( 128 ), tablename varchar( 128 ), textcolumn varchar( 128 ) ) parameter style java modifies sql data language java external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.createIndex'; create procedure LuceneSupport.dropIndex ( schemaname varchar( 128 ), tablename varchar( 128 ), textcolumn varchar( 128 ) ) parameter style java modifies sql data language java external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.dropIndex'; create procedure LuceneSupport.updateIndex ( schemaname varchar( 128 ), tablename varchar( 128 ), textcolumn varchar( 128 ) ) parameter style java reads sql data language java external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.updateIndex'; create function LuceneSupport.listIndexes() returns table ( id int, schemaname char( 128 ), tablename char( 128 ), columnname char( 128 ), lastupdated timestamp ) language java parameter style DERBY_JDBC_RESULT_SET contains sql external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.listIndexes' The LUCENETEST user can then index a text column as follows. The DBO can do this too. However, no-one else can index the column. That is because the createIndex() procedure attempts to create a table function in the lucenetest schema. Other users do not enjoy that privilege... call LuceneSupport.createIndex( 'lucenetest', 'titles', 'title' ); ...which creates the following directory structure... databaseName seg0 log lucene LUCENETEST TITLES TITLE ... ...and the following column-specific table function: create function lucenetest.titles__title( query varchar( 32672 ), rankCutoff double ) returns table ( ID int, documentID int, rank double ) language java parameter style derby_jdbc_result_set contains sql external name 'org.apache.derby.impl.optional.lucene.LuceneSupport.luceneQuery'; The LUCENETEST user can then query the Lucene index and join it to the original table as follows... select title, author, publisher, documentID, rank from lucenetest.titles t, table ( lucenetest.titles__title( 'grapes', 0 ) ) l where t.id = l.id; The LUCENETEST user (or the DBO but no-one else) can drop the Lucene index as follows... call LuceneSupport.dropIndex( 'lucenetest', 'titles','title' ); ...which drops the lucene/LUCENETEST/TITLES/TITLE directory. Directory deletion cascades up. That is, if there are no more indexed columns in lucenetest.titles, then we also drop lucene/LUCENETEST/TITLES. And we drop lucene/LUCENETEST if there are no more indexed tables in the lucenetest schema. The DBO (and only the DBO) can unload Lucene support as follows... call syscs_util.syscs_register_tool( 'luceneSupport', false ); ...which drops all procedures and table functions created by the tool. This command also drops the lucene subdirectory and everything under it. Touches the following files: -------------------- M java/engine/org/apache/derby/impl/sql/compile/StaticMethodCallNode.java Adds the Lucene support package to the list of Derby packages whose entry points can be bound to user-defined routines. -------------------- A java/engine/org/apache/derby/impl/optional A java/engine/org/apache/derby/impl/optional/lucene A java/engine/org/apache/derby/impl/optional/lucene/LuceneQueryVTI.java A java/engine/org/apache/derby/impl/optional/lucene/LuceneSupport.java A java/engine/org/apache/derby/impl/optional/lucene/LuceneListIndexesVTI.java A java/engine/org/apache/derby/impl/optional/lucene/build.xml M java/engine/org/apache/derby/impl/build.xml M java/engine/org/apache/derby/catalog/Java5SystemProcedures.java The Lucene support optional tool. -------------------- M java/engine/org/apache/derby/vti/VTITemplate.java Some additional support for context-aware table functions. This support allows the function to query the JDBC metadata for the shape of the ResultSet it returns. -------------------- M java/engine/org/apache/derby/loc/messages.xml M java/shared/org/apache/derby/shared/common/reference/SQLState.java New error messages for Lucene support. -------------------- M build.xml A tools/java/lucene-analyzers-common-4.5.0.jar A tools/java/lucene-queryparser-4.5.0.jar A tools/java/lucene-core-4.5.0.jar M tools/ant/properties/extrapath.properties M tools/jar/extraDBMSclasses.properties Miscellaneous build machinery. -------------------- A tools/release/notices/lucene.txt The Lucene notice file which must be included in the Derby NOTICE file if we are to ship the lucene jars in Derby source distributions. -------------------- M java/testing/org/apache/derbyTesting/functionTests/tests/lang/_Suite.java A java/testing/org/apache/derbyTesting/functionTests/tests/lang/LuceneSupportTest.java Initial tests.
          Hide
          Rick Hillegas added a comment -

          We have identified several issues with the current patch for Lucene support:

          1) It is not protected by SQL authorization.

          2) It is not protected by database encryption.

          3) It is not protected by backup/restore.

          4) It does not work with in-memory databases.

          In a follow-on comment, I will offer a revised patch and propose a solution to issue 1). But first I would like to describe an effort I made to store Lucene indexes in the database.

          Many issues with Lucene support could be solved if the Lucene index could be stored in the database. That would give us 1-4 for free. I searched for an off-the-shelf solution and I came across many pointers to a bundle of freeware called JdbcDirectory: http://www.compass-project.org/docs/2.2.0/reference/html/jdbcdirectory.html. This freeware provides an alternative implementation of the Lucene Directory interface, allowing you to represent indexes as Blobs.

          JdbcDirectory appears to be an abandoned effort. It must have worked with an older version of Lucene, but it does not run out-of-the-box with Lucene 4.5.0. I was not able to re-compile JdbcDirectory against Lucene 4.5.0. I think that some effort is needed to revive the JdbcDirectory solution.

          Other pointers suggested that JdbcDirectory performs poorly. So even after reviving this solution, it would address issues 1-4 at the cost of a significant performance hit.

          There may be other solutions to the problem of storing Lucene indexes in the database. However, I suspect that they all suffer from the fact that Lucene is a moving target. Lucene does not make the extensive backward-compatibility guarantees which Derby does. I have come to the following conclusions:

          1) Any effort to store Lucene indexes in the database is likely to break when customers upgrade their Lucene installation in order to take advantage of improvements to Lucene.

          2) As we pursue the approach begun in Andrew's patch, we should remember to document this warning: When you upgrade your Lucene installation, make sure you unload and then reload the Lucene support tool.

          Show
          Rick Hillegas added a comment - We have identified several issues with the current patch for Lucene support: 1) It is not protected by SQL authorization. 2) It is not protected by database encryption. 3) It is not protected by backup/restore. 4) It does not work with in-memory databases. In a follow-on comment, I will offer a revised patch and propose a solution to issue 1). But first I would like to describe an effort I made to store Lucene indexes in the database. Many issues with Lucene support could be solved if the Lucene index could be stored in the database. That would give us 1-4 for free. I searched for an off-the-shelf solution and I came across many pointers to a bundle of freeware called JdbcDirectory: http://www.compass-project.org/docs/2.2.0/reference/html/jdbcdirectory.html . This freeware provides an alternative implementation of the Lucene Directory interface, allowing you to represent indexes as Blobs. JdbcDirectory appears to be an abandoned effort. It must have worked with an older version of Lucene, but it does not run out-of-the-box with Lucene 4.5.0. I was not able to re-compile JdbcDirectory against Lucene 4.5.0. I think that some effort is needed to revive the JdbcDirectory solution. Other pointers suggested that JdbcDirectory performs poorly. So even after reviving this solution, it would address issues 1-4 at the cost of a significant performance hit. There may be other solutions to the problem of storing Lucene indexes in the database. However, I suspect that they all suffer from the fact that Lucene is a moving target. Lucene does not make the extensive backward-compatibility guarantees which Derby does. I have come to the following conclusions: 1) Any effort to store Lucene indexes in the database is likely to break when customers upgrade their Lucene installation in order to take advantage of improvements to Lucene. 2) As we pursue the approach begun in Andrew's patch, we should remember to document this warning: When you upgrade your Lucene installation, make sure you unload and then reload the Lucene support tool.
          Hide
          Rick Hillegas added a comment -

          Thanks for the second rev of the patch, Andrew. I have a couple comments under the following topics:

          o Build
          o Localization
          o Security
          o Misc

          I applied the patch, copied the Lucene jar files into tools/java, and successfully ran LuceneSupportTest. In addition, I ran some more experiments with a script attached at the end of this comment.

          Thanks,
          -Rick

          -------------- BUILD ----------------

          This optional tool relies on the following Lucene jar files. The user documentation should mention that these are necessary in order to use this tool:

          A tools/java/lucene-analyzers-common-4.5.0.jar
          A tools/java/lucene-queryparser-4.5.0.jar
          A tools/java/lucene-core-4.5.0.jar

          I see that you intend to check the Lucene jar files into the Derby source tree so that this code is always built and tested--even though I also see that you can build Derby cleanly without the Lucene jar files. I agree with your decision. We have spent considerable effort making it possible for all developers to build and test the entire Derby product, and I'd like to stay with that model. I understand that the Lucene jar files are big. They are an order of magnitude larger than any other jar files checked into Derby. But I don't think that including them in our source distributions is burdensome.

          I'm not clear on Lucene's compatibility guarantees:

          B1) Is there some reason that you chose jar files from Lucene 4.5.0 rather than the latest 4.5.1 release?

          B2) Our documentation should state which versions of Lucene we support. We should make sure that we verify those claims during release testing.

          -------------- LOCALIZATION ----------------

          L1) LuceneListIndexesVTI and LuceneQueryVTI raise SQLExceptions with hard-coded error strings. These should probably be replaced with localizable error strings. To see how to do this with optional tools, you can look for the error handle OT_BadLoadUnloadArgs in ForeignDBViews.

          L2) In addition, when LuceneQueryVTI.getRawColumn() catches an IOException, it should use that IOException as the cause of the SQLException that's thrown afterward.

          -------------- SECURITY ----------------

          I think that this tool should run comfortably when a Java security manager is installed. I recommend the following changes:

          S1) LuceneSupport.dropIndex() should do its File operations inside privilege blocks.

          S2) LuceneSupportTest should not disable the security manager.

          We need to give some thought to the SQL permissions needed to use this Lucene support. I have observed the following:

          S3) Only the DBO can load/unload this tool. That seems fine to me.

          S4) Ordinary users don't have permission to create/drop indexes. Is that too restrictive? It would be friendlier if anyone could create/drop indexes on any column they own. That is, on any column in a table in a schema that they own.

          S5) Ordinary users don't have permission to use the luceneQuery() table function.

          S6) ... but if EXECUTE privilege on luceneQuery() is granted to PUBLIC, then people can view any indexed data. This seems overbroad. If Fred doesn't have SELECT privilege on lucenetest.titles.title, then Fred shouldn't be able to see the contents of that column by running luceneQuery(). A poor man's solution to this would involve making luceneQuery() issue a dummy select against the indexed column first, just to make sure that the user has permission to see that data.

          S7) I have some misgivings about how this Lucene support will play with encrypted databases. The conglomerates will be encrypted but a great deal of information will leak out if the Lucene indexes are not encrypted. At a minimum, we should file a follow-on JIRA to collect thoughts/solutions related to this vulnerability. If we don't address this vulnerability, then we should at least describe it in our user documentation.

          -------------- MISC ----------------

          M1) LuceneListIndexesVTI and LuceneSupportTest need Apache license headers.

          M2) LuceneQueryVTI imports RestrictedVTI but doesn't implement it. Are you planning to make LuceneQueryVTI a RestrictedVTI in a future rev?

          M3) I like the isolation of the Lucene indexes in a separate "lucene" subdirectory of the database. I see that createIndex() creates a subdirectory with a name made out of the schema, table, and column arguments. There are probably some tricky edge cases involving delimited identifiers and maybe some security vulnerabilities involved in exposing these names. The edge cases and hypothetical security issues might be eliminated by giving these subdirectories less friendly names like $tableID_$columnNumber.

          -------------- TEST SCRIPT ----------------

          connect 'jdbc:derby:db1;create=true;user=dbo';

          call syscs_util.syscs_create_user( 'DBO', 'dbo_password' );
          call syscs_util.syscs_create_user( 'LUCENETEST', 'lucenetest_password' );
          call syscs_util.syscs_create_user( 'FRED', 'fred_password' );

          – shutdown in order to enable NATIVE authentication
          connect 'jdbc:derby:db1;shutdown=true';

          connect 'jdbc:derby:db1;user=lucenetest;password=lucenetest_password' as lucenetest;

          create table titles (ID int generated always as identity, ISBN varchar(16), PRINTISBN varchar(16), title varchar(1024), subtitle varchar(1024), author varchar(1024), series varchar(1024), publisher varchar(1024), collections varchar(128), collections2 varchar(128));

          insert into titles (ISBN, PRINTISBN, TITLE, SUBTITLE, AUTHOR, SERIES, PUBLISHER, COLLECTIONS, COLLECTIONS2) values ('9765087650324','9765087650324','The Grapes Of Wrath','The Great Depression in Oklahoma','John Steinbeck','Noble Winners','The Viking Press','National Book Award','Pulitzer Prize');

          insert into titles (ISBN, PRINTISBN, TITLE, SUBTITLE, AUTHOR, SERIES, PUBLISHER, COLLECTIONS, COLLECTIONS2) values ('6754278542987','6754278542987','Identical: Portraits of Twins','Best Photo Book 2012 by American Photo Magazine','Martin Schoeller','Portraits','teNeues','Photography','');

          insert into titles (ISBN, PRINTISBN, TITLE, SUBTITLE, AUTHOR, SERIES, PUBLISHER, COLLECTIONS, COLLECTIONS2) values ('2747583475882','2747583475882','Vines, Grapes, and Wines','The wine drinker''s guide to grape varieties','Jancis Robinson','Reference','Alfred A. Knopf','Wine','');

          insert into titles (ISBN, PRINTISBN, TITLE, SUBTITLE, AUTHOR, SERIES, PUBLISHER, COLLECTIONS, COLLECTIONS2) values ('4356123483483','4356123483483','A Tale of Two Cities','A fictional account of events leading up to the French revolution','Charles Dickens','Classics','Chapman & Hall','Fiction','Social Criticism');

          – as expected, this user doesn't have permission to load the tool
          call syscs_util.syscs_register_tool('luceneSupport',true);

          connect 'jdbc:derby:db1;user=dbo;password=dbo_password' as dbo;

          – succeeds because dbo has permission to load optional tools
          call syscs_util.syscs_register_tool('luceneSupport',true);

          set connection lucenetest;

          – fails due to lack of permission. probably shouldn't fail.
          call LuceneSupport.createIndex('lucenetest','titles','title','id');

          set connection dbo;

          – succeeds
          call LuceneSupport.createIndex('lucenetest','titles','title','id');

          set connection lucenetest;

          – fails due to lack of permission. probably shouldn't fail.
          select * from table ( LuceneSupport.luceneQuery('grapes', 'lucenetest.titles', 'title', 0) ) luceneResults;

          set connection dbo;

          – succeeds
          select * from table ( LuceneSupport.luceneQuery('grapes', 'lucenetest.titles', 'title', 0) ) luceneResults;

          grant execute on function LuceneSupport.luceneQuery to public;

          set connection lucenetest;

          – works now, after dbo lets everyone run the luceneQuery table function
          select * from table ( LuceneSupport.luceneQuery('grapes', 'lucenetest.titles', 'title', 0) ) luceneResults;

          connect 'jdbc:derby:db1;user=fred;password=fred_password' as dbo;

          – it's unfortunate that fred can view data in a table he doesn't have SELECT access to
          select * from table ( LuceneSupport.luceneQuery('grapes', 'lucenetest.titles', 'title', 0) ) luceneResults;

          – as expected, the following query fails due to lack of permission
          select title from lucenetest.titles where 1=2;

          set connection dbo;

          select id, schemaname, tablename, columnname from table ( LuceneSupport.listIndexes('', '') ) listindexes;

          Show
          Rick Hillegas added a comment - Thanks for the second rev of the patch, Andrew. I have a couple comments under the following topics: o Build o Localization o Security o Misc I applied the patch, copied the Lucene jar files into tools/java, and successfully ran LuceneSupportTest. In addition, I ran some more experiments with a script attached at the end of this comment. Thanks, -Rick -------------- BUILD ---------------- This optional tool relies on the following Lucene jar files. The user documentation should mention that these are necessary in order to use this tool: A tools/java/lucene-analyzers-common-4.5.0.jar A tools/java/lucene-queryparser-4.5.0.jar A tools/java/lucene-core-4.5.0.jar I see that you intend to check the Lucene jar files into the Derby source tree so that this code is always built and tested--even though I also see that you can build Derby cleanly without the Lucene jar files. I agree with your decision. We have spent considerable effort making it possible for all developers to build and test the entire Derby product, and I'd like to stay with that model. I understand that the Lucene jar files are big. They are an order of magnitude larger than any other jar files checked into Derby. But I don't think that including them in our source distributions is burdensome. I'm not clear on Lucene's compatibility guarantees: B1) Is there some reason that you chose jar files from Lucene 4.5.0 rather than the latest 4.5.1 release? B2) Our documentation should state which versions of Lucene we support. We should make sure that we verify those claims during release testing. -------------- LOCALIZATION ---------------- L1) LuceneListIndexesVTI and LuceneQueryVTI raise SQLExceptions with hard-coded error strings. These should probably be replaced with localizable error strings. To see how to do this with optional tools, you can look for the error handle OT_BadLoadUnloadArgs in ForeignDBViews. L2) In addition, when LuceneQueryVTI.getRawColumn() catches an IOException, it should use that IOException as the cause of the SQLException that's thrown afterward. -------------- SECURITY ---------------- I think that this tool should run comfortably when a Java security manager is installed. I recommend the following changes: S1) LuceneSupport.dropIndex() should do its File operations inside privilege blocks. S2) LuceneSupportTest should not disable the security manager. We need to give some thought to the SQL permissions needed to use this Lucene support. I have observed the following: S3) Only the DBO can load/unload this tool. That seems fine to me. S4) Ordinary users don't have permission to create/drop indexes. Is that too restrictive? It would be friendlier if anyone could create/drop indexes on any column they own. That is, on any column in a table in a schema that they own. S5) Ordinary users don't have permission to use the luceneQuery() table function. S6) ... but if EXECUTE privilege on luceneQuery() is granted to PUBLIC, then people can view any indexed data. This seems overbroad. If Fred doesn't have SELECT privilege on lucenetest.titles.title, then Fred shouldn't be able to see the contents of that column by running luceneQuery(). A poor man's solution to this would involve making luceneQuery() issue a dummy select against the indexed column first, just to make sure that the user has permission to see that data. S7) I have some misgivings about how this Lucene support will play with encrypted databases. The conglomerates will be encrypted but a great deal of information will leak out if the Lucene indexes are not encrypted. At a minimum, we should file a follow-on JIRA to collect thoughts/solutions related to this vulnerability. If we don't address this vulnerability, then we should at least describe it in our user documentation. -------------- MISC ---------------- M1) LuceneListIndexesVTI and LuceneSupportTest need Apache license headers. M2) LuceneQueryVTI imports RestrictedVTI but doesn't implement it. Are you planning to make LuceneQueryVTI a RestrictedVTI in a future rev? M3) I like the isolation of the Lucene indexes in a separate "lucene" subdirectory of the database. I see that createIndex() creates a subdirectory with a name made out of the schema, table, and column arguments. There are probably some tricky edge cases involving delimited identifiers and maybe some security vulnerabilities involved in exposing these names. The edge cases and hypothetical security issues might be eliminated by giving these subdirectories less friendly names like $tableID_$columnNumber. -------------- TEST SCRIPT ---------------- connect 'jdbc:derby:db1;create=true;user=dbo'; call syscs_util.syscs_create_user( 'DBO', 'dbo_password' ); call syscs_util.syscs_create_user( 'LUCENETEST', 'lucenetest_password' ); call syscs_util.syscs_create_user( 'FRED', 'fred_password' ); – shutdown in order to enable NATIVE authentication connect 'jdbc:derby:db1;shutdown=true'; connect 'jdbc:derby:db1;user=lucenetest;password=lucenetest_password' as lucenetest; create table titles (ID int generated always as identity, ISBN varchar(16), PRINTISBN varchar(16), title varchar(1024), subtitle varchar(1024), author varchar(1024), series varchar(1024), publisher varchar(1024), collections varchar(128), collections2 varchar(128)); insert into titles (ISBN, PRINTISBN, TITLE, SUBTITLE, AUTHOR, SERIES, PUBLISHER, COLLECTIONS, COLLECTIONS2) values ('9765087650324','9765087650324','The Grapes Of Wrath','The Great Depression in Oklahoma','John Steinbeck','Noble Winners','The Viking Press','National Book Award','Pulitzer Prize'); insert into titles (ISBN, PRINTISBN, TITLE, SUBTITLE, AUTHOR, SERIES, PUBLISHER, COLLECTIONS, COLLECTIONS2) values ('6754278542987','6754278542987','Identical: Portraits of Twins','Best Photo Book 2012 by American Photo Magazine','Martin Schoeller','Portraits','teNeues','Photography',''); insert into titles (ISBN, PRINTISBN, TITLE, SUBTITLE, AUTHOR, SERIES, PUBLISHER, COLLECTIONS, COLLECTIONS2) values ('2747583475882','2747583475882','Vines, Grapes, and Wines','The wine drinker''s guide to grape varieties','Jancis Robinson','Reference','Alfred A. Knopf','Wine',''); insert into titles (ISBN, PRINTISBN, TITLE, SUBTITLE, AUTHOR, SERIES, PUBLISHER, COLLECTIONS, COLLECTIONS2) values ('4356123483483','4356123483483','A Tale of Two Cities','A fictional account of events leading up to the French revolution','Charles Dickens','Classics','Chapman & Hall','Fiction','Social Criticism'); – as expected, this user doesn't have permission to load the tool call syscs_util.syscs_register_tool('luceneSupport',true); connect 'jdbc:derby:db1;user=dbo;password=dbo_password' as dbo; – succeeds because dbo has permission to load optional tools call syscs_util.syscs_register_tool('luceneSupport',true); set connection lucenetest; – fails due to lack of permission. probably shouldn't fail. call LuceneSupport.createIndex('lucenetest','titles','title','id'); set connection dbo; – succeeds call LuceneSupport.createIndex('lucenetest','titles','title','id'); set connection lucenetest; – fails due to lack of permission. probably shouldn't fail. select * from table ( LuceneSupport.luceneQuery('grapes', 'lucenetest.titles', 'title', 0) ) luceneResults; set connection dbo; – succeeds select * from table ( LuceneSupport.luceneQuery('grapes', 'lucenetest.titles', 'title', 0) ) luceneResults; grant execute on function LuceneSupport.luceneQuery to public; set connection lucenetest; – works now, after dbo lets everyone run the luceneQuery table function select * from table ( LuceneSupport.luceneQuery('grapes', 'lucenetest.titles', 'title', 0) ) luceneResults; connect 'jdbc:derby:db1;user=fred;password=fred_password' as dbo; – it's unfortunate that fred can view data in a table he doesn't have SELECT access to select * from table ( LuceneSupport.luceneQuery('grapes', 'lucenetest.titles', 'title', 0) ) luceneResults; – as expected, the following query fails due to lack of permission select title from lucenetest.titles where 1=2; set connection dbo; select id, schemaname, tablename, columnname from table ( LuceneSupport.listIndexes('', '') ) listindexes;
          Hide
          Rick Hillegas added a comment -

          Thanks for the latest increment, Andrew. I plan to take a look at this soon. Thanks.

          Show
          Rick Hillegas added a comment - Thanks for the latest increment, Andrew. I plan to take a look at this soon. Thanks.
          Hide
          Andrew McIntyre added a comment -

          Reattaching lucene_demo_2.diff, this time with the rankCutoff part of the query function working and additional tests for same.

          Show
          Andrew McIntyre added a comment - Reattaching lucene_demo_2.diff, this time with the rankCutoff part of the query function working and additional tests for same.
          Hide
          Andrew McIntyre added a comment -

          Attached a second patch which now includes javadoc, tightens up the comments, and includes a starter set of tests for the functionality. Still need to dig up a fuller set of data, I'll try to get to that maybe this weekend. In the meantime, I made some up with some things I had on my bookshelf and the schema from before.

          • indexTable() in lucene_titles.sql was also from a previous experiment, I had changed it to luceneIndex() for the patch in lucene_demo.diff, but for the updated patch in lucene_demo_2.diff, I decided to standardize the naming with createIndex, updateIndex, dropIndex, etc.

          -The indexes are now hopefully all located in the database directory, per-db, and I liked the suggestion for listing out the indexes, so also included is another function/VTI for listing out the Lucene indexes for the current database.

          -More tests and documentation to come. At some point I'll try to get the rankCutoff function working as well.

          Show
          Andrew McIntyre added a comment - Attached a second patch which now includes javadoc, tightens up the comments, and includes a starter set of tests for the functionality. Still need to dig up a fuller set of data, I'll try to get to that maybe this weekend. In the meantime, I made some up with some things I had on my bookshelf and the schema from before. indexTable() in lucene_titles.sql was also from a previous experiment, I had changed it to luceneIndex() for the patch in lucene_demo.diff, but for the updated patch in lucene_demo_2.diff, I decided to standardize the naming with createIndex, updateIndex, dropIndex, etc. -The indexes are now hopefully all located in the database directory, per-db, and I liked the suggestion for listing out the indexes, so also included is another function/VTI for listing out the Lucene indexes for the current database. -More tests and documentation to come. At some point I'll try to get the rankCutoff function working as well.
          Hide
          Rick Hillegas added a comment -

          Thanks for that additional information, Andrew. Concerning i, here are some thoughts:

          1) I see that this new additional tool is being parked next to the existing DBMDWrapper and ForeignDBViews classes. I do agree that all of these tools should be neighbors. Just a heads-up, however: I think I put DBMDWrapper and ForeignDBViews in the wrong jar file to begin with. I now think those tools really ought to go into the engine jar, for reasons which I gave in a 2013-06-13 comment on https://issues.apache.org/jira/browse/DERBY-6256. To summarize: the point of derbytools.jar is to hold code which runs both client-side and server-side. None of these tools run client-side. My point is this: I'm not faulting the patch for following the existing pattern; instead, I'm saying that I put the original tools in the wrong place. And at some point I may file a follow-on JIRA to move all of these tools into the engine jar.

          2) If I understand the patch correctly, the indexTable() procedure really indexes a column. You could run the procedure multiple times on the same table in order to index different columns. I think that createDocumentIndex() would be a better name for this procedure.

          3) Similarly, I'm not keen on the name luceneUpdateDocument() for several reasons. I think that the "lucene" prefix can be assumed from the name of the schema which holds this procedure. I'd also like the procedure names to express the fact that luceneUpdateDocument() refreshes the index created by indexTable(). So I'd recommend something like updateDocumentIndex(), akin to createDocumentIndex().

          Some more thoughts follow:

          4) In my experience, every insert function needs to be matched by corresponding update and delete functions. Developers expect that. This tool provides an insert function (createDocumentIndex()) and a corresponding update function (updateDocumentIndex()) but no corresponding delete function. The delete function is really important as developers hack out their schemas in the laboratory. So I recommend adding a dropDocumentIndex() procedure. I understand your reservations about deleting a whole directory, but I think that the first enhancement request we'll get is "give me a way to delete these things."

          5) The non-transactional behavior of these procedures needs to be clearly understood by users. That's probably a documentation issue. But users need to understand that they can't rollback some of the important effects of calls to createDocumentIndex() and dropDocumentIndex().

          6) Developers hacking out a schema in the laboratory will also want tools for introspecting which columns have been indexed and how current the indexes are. Maybe the best solution would be a view wrapping a table function which, in turn, exposes metadata from Lucene and/or the file system. If that's not possible, a table like the following could be useful:

          LuceneSupport.documentIndexes
          (
          id int generated always as identity,
          tableID char( 36 ) not null,
          columnNumber int,
          lastupdated timestamp,
          unique( tableID, columnNumber )
          );

          I prefer the table function solution because it makes it harder for the user to mess up and accidentally delete this metadata.

          7) At this point, I don't see a need for the syntactic sugar of new SQL statements. I think that the optional tool approach is fine for this first increment. I recommend deferring any parser work until later, after we've cleared up the transactional consistency issues.

          Thanks!
          -Rick

          Show
          Rick Hillegas added a comment - Thanks for that additional information, Andrew. Concerning i, here are some thoughts: 1) I see that this new additional tool is being parked next to the existing DBMDWrapper and ForeignDBViews classes. I do agree that all of these tools should be neighbors. Just a heads-up, however: I think I put DBMDWrapper and ForeignDBViews in the wrong jar file to begin with. I now think those tools really ought to go into the engine jar, for reasons which I gave in a 2013-06-13 comment on https://issues.apache.org/jira/browse/DERBY-6256 . To summarize: the point of derbytools.jar is to hold code which runs both client-side and server-side. None of these tools run client-side. My point is this: I'm not faulting the patch for following the existing pattern; instead, I'm saying that I put the original tools in the wrong place. And at some point I may file a follow-on JIRA to move all of these tools into the engine jar. 2) If I understand the patch correctly, the indexTable() procedure really indexes a column. You could run the procedure multiple times on the same table in order to index different columns. I think that createDocumentIndex() would be a better name for this procedure. 3) Similarly, I'm not keen on the name luceneUpdateDocument() for several reasons. I think that the "lucene" prefix can be assumed from the name of the schema which holds this procedure. I'd also like the procedure names to express the fact that luceneUpdateDocument() refreshes the index created by indexTable(). So I'd recommend something like updateDocumentIndex(), akin to createDocumentIndex(). Some more thoughts follow: 4) In my experience, every insert function needs to be matched by corresponding update and delete functions. Developers expect that. This tool provides an insert function (createDocumentIndex()) and a corresponding update function (updateDocumentIndex()) but no corresponding delete function. The delete function is really important as developers hack out their schemas in the laboratory. So I recommend adding a dropDocumentIndex() procedure. I understand your reservations about deleting a whole directory, but I think that the first enhancement request we'll get is "give me a way to delete these things." 5) The non-transactional behavior of these procedures needs to be clearly understood by users. That's probably a documentation issue. But users need to understand that they can't rollback some of the important effects of calls to createDocumentIndex() and dropDocumentIndex(). 6) Developers hacking out a schema in the laboratory will also want tools for introspecting which columns have been indexed and how current the indexes are. Maybe the best solution would be a view wrapping a table function which, in turn, exposes metadata from Lucene and/or the file system. If that's not possible, a table like the following could be useful: LuceneSupport.documentIndexes ( id int generated always as identity, tableID char( 36 ) not null, columnNumber int, lastupdated timestamp, unique( tableID, columnNumber ) ); I prefer the table function solution because it makes it harder for the user to mess up and accidentally delete this metadata. 7) At this point, I don't see a need for the syntactic sugar of new SQL statements. I think that the optional tool approach is fine for this first increment. I recommend deferring any parser work until later, after we've cleared up the transactional consistency issues. Thanks! -Rick
          Hide
          Andrew McIntyre added a comment -

          Looks like I grabbed some extra text when copying things over to my Apache account. indexDatabase() was from an earlier experiment and can be ignored. I've removed it from lucene_titles.sql.

          for iii - I think SQL exceptions in case of missing / corrupted indexes will do for now, as if another index were missing or corrupted. My original idea for organizing the indexes was to keep the indexes in a per-db 'lucene' directory at the same level as 'log' and 'seg0', then in subdirectories named 'schema_table_column,' so it would be obvious looking in the db's lucene directory which columns were indexed. That's not what we have here, which is a system-wide lucene directory instead of a per-db directory. I'll try to get a per-db dir for the next patch, if I can pry the db directory out of the monitor/store. With a per-db lucene dir, if the database directory is backed up, the lucene indexes would be be backed up along with also. I also thought about having a drop lucene index procedure, but figured for the moment leaving that to the user/developer, since it involves deleting directories in the filesystem

          ii/iv - comments/javadoc/tests to come in the next patch. Main problem right now is finding time, and scraping/generating a decently large set of test data that is ok to reuse. The lucene project probably has some, though.

          as for i - feel free to let me know any particulars, or you can wait for the next patch. it will probably be a few days.

          Show
          Andrew McIntyre added a comment - Looks like I grabbed some extra text when copying things over to my Apache account. indexDatabase() was from an earlier experiment and can be ignored. I've removed it from lucene_titles.sql. for iii - I think SQL exceptions in case of missing / corrupted indexes will do for now, as if another index were missing or corrupted. My original idea for organizing the indexes was to keep the indexes in a per-db 'lucene' directory at the same level as 'log' and 'seg0', then in subdirectories named 'schema_table_column,' so it would be obvious looking in the db's lucene directory which columns were indexed. That's not what we have here, which is a system-wide lucene directory instead of a per-db directory. I'll try to get a per-db dir for the next patch, if I can pry the db directory out of the monitor/store. With a per-db lucene dir, if the database directory is backed up, the lucene indexes would be be backed up along with also. I also thought about having a drop lucene index procedure, but figured for the moment leaving that to the user/developer, since it involves deleting directories in the filesystem ii/iv - comments/javadoc/tests to come in the next patch. Main problem right now is finding time, and scraping/generating a decently large set of test data that is ok to reuse. The lucene project probably has some, though. as for i - feel free to let me know any particulars, or you can wait for the next patch. it will probably be a few days.
          Hide
          Rick Hillegas added a comment -

          Thanks again for working on this, Andrew. I noticed that lucene_titles.sql invokes a procedure called LuceneSupport.indexDatabase(). I can't find that procedure in lucene_demo.diff. Where should I look for that procedure?

          Here's my crude interpretation of what the code is doing: The tool makes it possible to do full-text search on data which is stored in the text columns of Derby tables. The tables must have unique Derby indexes. Lucene itself relies on indexes which it builds and stores outside Derby in the file system. Over time, the Lucene indexes drift out of sync with the text data. The application periodically asks Derby to update specific Lucene indexes, bringing them back into sync with the text data.

          Loading the tool via syscs_register_tool() creates the following schema objects:

          a) LuceneSupport.indexTable() - This procedure indexes a text column in a Derby table.

          b) LuceneSupport.luceneUpdateDocument() - This procedure updates a Lucene index which was created by the previous procedure, bringing the Lucene index back into sync with the text data.

          c) LuceneSupport.luceneQuery() - This is a table function for running a full-text search against a Derby column.

          As is, this sounds like a very useful piece of functionality. We could make this production-ready incrementally and document it at the end of that effort. At a minimum, we would want to:

          i) Quibble a bit about the api, the names of schema objects, and where the code goes.

          ii) Add comments to the code.

          iii) Think about edge cases. For example, what happens if the Lucene indexes become corrupt or are deleted? How do we keep track of which columns are indexed? What happens when Derby is recovered from a backup or the database is recreated?

          iv) Write tests.

          Some follow-on efforts might also make sense:

          1) We could consider moving the Lucene indexes inside the database.

          2) Maybe we could add triggers on the indexed columns so that the Lucene indexes remain in sync with the Derby data. Don't know how much of a performance drag that would be. Maybe this could be an optional feature of creating a Lucene index.

          3) Replace the procedure calls with explicit CREATE FULLTEXT (and maybe UPDATE FULLTEXT) statements. This would be an opportunity to think about how we could load and unload optional Derby statements.

          Thanks!
          -Rick

          Show
          Rick Hillegas added a comment - Thanks again for working on this, Andrew. I noticed that lucene_titles.sql invokes a procedure called LuceneSupport.indexDatabase(). I can't find that procedure in lucene_demo.diff. Where should I look for that procedure? Here's my crude interpretation of what the code is doing: The tool makes it possible to do full-text search on data which is stored in the text columns of Derby tables. The tables must have unique Derby indexes. Lucene itself relies on indexes which it builds and stores outside Derby in the file system. Over time, the Lucene indexes drift out of sync with the text data. The application periodically asks Derby to update specific Lucene indexes, bringing them back into sync with the text data. Loading the tool via syscs_register_tool() creates the following schema objects: a) LuceneSupport.indexTable() - This procedure indexes a text column in a Derby table. b) LuceneSupport.luceneUpdateDocument() - This procedure updates a Lucene index which was created by the previous procedure, bringing the Lucene index back into sync with the text data. c) LuceneSupport.luceneQuery() - This is a table function for running a full-text search against a Derby column. As is, this sounds like a very useful piece of functionality. We could make this production-ready incrementally and document it at the end of that effort. At a minimum, we would want to: i) Quibble a bit about the api, the names of schema objects, and where the code goes. ii) Add comments to the code. iii) Think about edge cases. For example, what happens if the Lucene indexes become corrupt or are deleted? How do we keep track of which columns are indexed? What happens when Derby is recovered from a backup or the database is recreated? iv) Write tests. Some follow-on efforts might also make sense: 1) We could consider moving the Lucene indexes inside the database. 2) Maybe we could add triggers on the indexed columns so that the Lucene indexes remain in sync with the Derby data. Don't know how much of a performance drag that would be. Maybe this could be an optional feature of creating a Lucene index. 3) Replace the procedure calls with explicit CREATE FULLTEXT (and maybe UPDATE FULLTEXT) statements. This would be an opportunity to think about how we could load and unload optional Derby statements. Thanks! -Rick
          Hide
          Andrew McIntyre added a comment -

          Attached patch sent to mailing list. ij script that can be used to demonstrate the patch's features is here:

          http://people.apache.org/~fuzzylogic/lucene_titles.sql

          although this contains what might be considered copyrighted data, so this will be removed and I'll try to replace it with a more appropriate dataset and include it directly in the patch when I can find one that works well. As I said, very rough, I basically just got it working. Some things don't work (the rankCutoff parameter in the query function doesn't do anything), it needs formatting, comments, javadoc, tests, etc. etc.

          Regarding Rick's comment on the dev list about further integration via the parser, it seems like it would be pretty straightforward to go from the procedure call in this patch to wiring the index creation into the sql grammar by using the MySQL-ish: CREATE FULLTEXT INDEX INDEX_NAME ON SCHEMA.TABLE (COLUMN)

          Show
          Andrew McIntyre added a comment - Attached patch sent to mailing list. ij script that can be used to demonstrate the patch's features is here: http://people.apache.org/~fuzzylogic/lucene_titles.sql although this contains what might be considered copyrighted data, so this will be removed and I'll try to replace it with a more appropriate dataset and include it directly in the patch when I can find one that works well. As I said, very rough, I basically just got it working. Some things don't work (the rankCutoff parameter in the query function doesn't do anything), it needs formatting, comments, javadoc, tests, etc. etc. Regarding Rick's comment on the dev list about further integration via the parser, it seems like it would be pretty straightforward to go from the procedure call in this patch to wiring the index creation into the sql grammar by using the MySQL-ish: CREATE FULLTEXT INDEX INDEX_NAME ON SCHEMA.TABLE (COLUMN)
          Hide
          Andrew McIntyre added a comment -

          Attaching rough draft of patch for optional tool to create, update, and query Lucene indexes in Derby.

          Show
          Andrew McIntyre added a comment - Attaching rough draft of patch for optional tool to create, update, and query Lucene indexes in Derby.
          Hide
          Rick Hillegas added a comment -

          Linking this issue to a later request for Lucene integration.

          Show
          Rick Hillegas added a comment - Linking this issue to a later request for Lucene integration.
          Hide
          scott hutinger added a comment -

          http://issues.apache.org/jira/browse/LUCENE-434

          From the Desciption:
          "Code and examples for embedding Lucene in HSQLDB and Derby relational databases."

          This would more than likely be a good place to get a bit of information.

          Show
          scott hutinger added a comment - http://issues.apache.org/jira/browse/LUCENE-434 From the Desciption: "Code and examples for embedding Lucene in HSQLDB and Derby relational databases." This would more than likely be a good place to get a bit of information.

            People

            • Assignee:
              Unassigned
              Reporter:
              Abhijeet Mahesh
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development