Derby
  1. Derby
  2. DERBY-6564

Document the experimental, optional LuceneSupport tool.

    Details

    • Urgency:
      Normal

      Description

      The LucenePlugin.html functional spec attached to DERBY-590 has been updated. This feature is ready to be documented. Thanks.

      1. DERBY-6564-3.zip
        11 kB
        Kim Haase
      2. DERBY-6564-3.stat
        0.2 kB
        Kim Haase
      3. DERBY-6564-3.diff
        8 kB
        Kim Haase
      4. DERBY-6564-upgrade.diff
        0.9 kB
        Kim Haase
      5. rtoolsoptlucene.html
        13 kB
        Kim Haase
      6. DERBY-6564-2.zip
        19 kB
        Kim Haase
      7. DERBY-6564-2.diff
        35 kB
        Kim Haase
      8. DERBY-6564.zip
        19 kB
        Kim Haase
      9. DERBY-6564.stat
        0.3 kB
        Kim Haase
      10. DERBY-6564.diff
        35 kB
        Kim Haase

        Issue Links

          Activity

          Hide
          Kim Haase added a comment -

          The functional spec is for the most part really clear – thanks, Rick. I have a few questions here and there, though:

          Under "Terms" –

          score – what are the minimum and maximum values? is it like a percent (0 to 100)? or 0 to 1.0?

          keycolumns – comma-separated list, I gather?

          Under "Querying an index", a line in the returned table says

          DOCUMENT ID INT,

          This is supposed to be

          DOCUMENTID INT,

          isn't it?

          I thought with a score the higher values indicated a better match. So why would you only want to return results with scores under a certain value (SCORECEILING)? And what would specifying a SCORECEILING of 0 do? I thought it would produce no results.

          Under "Classpath" – Where do you get the Lucene jars?

          Thanks!

          Show
          Kim Haase added a comment - The functional spec is for the most part really clear – thanks, Rick. I have a few questions here and there, though: Under "Terms" – score – what are the minimum and maximum values? is it like a percent (0 to 100)? or 0 to 1.0? keycolumns – comma-separated list, I gather? Under "Querying an index", a line in the returned table says DOCUMENT ID INT, This is supposed to be DOCUMENTID INT, isn't it? I thought with a score the higher values indicated a better match. So why would you only want to return results with scores under a certain value (SCORECEILING)? And what would specifying a SCORECEILING of 0 do? I thought it would produce no results. Under "Classpath" – Where do you get the Lucene jars? Thanks!
          Hide
          Kim Haase added a comment -

          It seems to be slightly inconsistent that although we document 3 optional tools (databaseMetaData and foreignViews are the other two), only the Lucene one is in the new derbyoptionaltools.jar file. Is there any thought of putting the other two there as well? They are currently in derbytools.jar. On the other hand, it would be disruptive to move them ...

          Show
          Kim Haase added a comment - It seems to be slightly inconsistent that although we document 3 optional tools (databaseMetaData and foreignViews are the other two), only the Lucene one is in the new derbyoptionaltools.jar file. Is there any thought of putting the other two there as well? They are currently in derbytools.jar. On the other hand, it would be disruptive to move them ...
          Hide
          Rick Hillegas added a comment -

          Hi Kim,

          Thanks for asking these questions. They have caused me to think a little more deeply about the meaning of score and scoreCeiling. I think I need to change the api a little.

          CH> score – what are the minimum and maximum values? is it like a percent (0 to 100)? or 0 to 1.0?
          CH>

          I don't think there is a minimum or maximum score. The only contract is that higher scores indicate a better match. Some googling suggest that scores can even be negative. See, for instance, http://stackoverflow.com/questions/8918224/what-is-the-maximum-value-of-a-lucene-score

          CH> keycolumns – comma-separated list, I gather?
          CH>

          Right. Its a vararg, a variable length (possibly empty) comma-separated list.

          CH> Under "Querying an index", a line in the returned table says
          CH> DOCUMENT ID INT,
          CH> This is supposed to be
          CH> DOCUMENTID INT,
          CH> isn't it?
          CH>

          Right. Sorry about that.

          CH> I thought with a score the higher values indicated a better
          CH> match. So why would you only want to return results with scores under
          CH> a certain value (SCORECEILING)?

          This is how you page through blocks of Lucene results. Let's compare this to the SQL LIMIT/OFFSET clause. In SQL, you can use LIMIT/OFFSET to say "start at the nth row and give me the next m results". With Lucene you say "start at score n and give me the next m results".

          CH> And what would specifying a SCORECEILING of 0 do? I thought it would produce no results.
          CH>

          I need to fix this. Right now the plugin treats 0 as a special value meaning "no scoreCeiling, start from the top". I want to change this so that 0 is just another score value and, instead, use null to mean "no scoreCeiling, start from the top". Thanks for bringing this up!

          CH> Under "Classpath" – Where do you get the Lucene jars?

          A version of Lucene is checked into the Derby source tree. The three jars you need are in tools/java. They are the 4.7.1 Lucene jars. You can get other versions from the Lucene website at http://lucene.apache.org/core/.

          Thanks!
          -Rick

          Show
          Rick Hillegas added a comment - Hi Kim, Thanks for asking these questions. They have caused me to think a little more deeply about the meaning of score and scoreCeiling. I think I need to change the api a little. CH> score – what are the minimum and maximum values? is it like a percent (0 to 100)? or 0 to 1.0? CH> I don't think there is a minimum or maximum score. The only contract is that higher scores indicate a better match. Some googling suggest that scores can even be negative. See, for instance, http://stackoverflow.com/questions/8918224/what-is-the-maximum-value-of-a-lucene-score CH> keycolumns – comma-separated list, I gather? CH> Right. Its a vararg, a variable length (possibly empty) comma-separated list. CH> Under "Querying an index", a line in the returned table says CH> DOCUMENT ID INT, CH> This is supposed to be CH> DOCUMENTID INT, CH> isn't it? CH> Right. Sorry about that. CH> I thought with a score the higher values indicated a better CH> match. So why would you only want to return results with scores under CH> a certain value (SCORECEILING)? This is how you page through blocks of Lucene results. Let's compare this to the SQL LIMIT/OFFSET clause. In SQL, you can use LIMIT/OFFSET to say "start at the nth row and give me the next m results". With Lucene you say "start at score n and give me the next m results". CH> And what would specifying a SCORECEILING of 0 do? I thought it would produce no results. CH> I need to fix this. Right now the plugin treats 0 as a special value meaning "no scoreCeiling, start from the top". I want to change this so that 0 is just another score value and, instead, use null to mean "no scoreCeiling, start from the top". Thanks for bringing this up! CH> Under "Classpath" – Where do you get the Lucene jars? A version of Lucene is checked into the Derby source tree. The three jars you need are in tools/java. They are the 4.7.1 Lucene jars. You can get other versions from the Lucene website at http://lucene.apache.org/core/ . Thanks! -Rick
          Hide
          Rick Hillegas added a comment -

          CH> It seems to be slightly inconsistent that although we document 3 optional tools (databaseMetaData and foreignViews are the other two), only the Lucene one is in the new derbyoptionaltools.jar file. Is there any thought of putting the other two there as well? They are currently in derbytools.jar. On the other hand, it would be disruptive to move them

          You're right. That looks pretty goofy. I've filed DERBY-6570 to track this.

          Thanks!
          -Rick

          Show
          Rick Hillegas added a comment - CH> It seems to be slightly inconsistent that although we document 3 optional tools (databaseMetaData and foreignViews are the other two), only the Lucene one is in the new derbyoptionaltools.jar file. Is there any thought of putting the other two there as well? They are currently in derbytools.jar. On the other hand, it would be disruptive to move them You're right. That looks pretty goofy. I've filed DERBY-6570 to track this. Thanks! -Rick
          Hide
          Kim Haase added a comment -

          Thanks for all the great explanations, Rick. This is very helpful.

          I'll keep plugging along and stay tuned for updates.

          Show
          Kim Haase added a comment - Thanks for all the great explanations, Rick. This is very helpful. I'll keep plugging along and stay tuned for updates.
          Hide
          Rick Hillegas added a comment - - edited

          Hi Kim,

          I have just committed derby-590-30-aa-nullableScoreCeiling.diff. This makes the change noted above:

          Concerning the values of the scoreCeiling argument to the Lucene query table functions: NULL (rather than 0) means "no score ceiling".

          I've also attached a new rev of the functional spec to DERBY-590, clarifying this point.

          Thanks again for noticing this.

          Show
          Rick Hillegas added a comment - - edited Hi Kim, I have just committed derby-590-30-aa-nullableScoreCeiling.diff. This makes the change noted above: Concerning the values of the scoreCeiling argument to the Lucene query table functions: NULL (rather than 0) means "no score ceiling". I've also attached a new rev of the functional spec to DERBY-590 , clarifying this point. Thanks again for noticing this.
          Hide
          Kim Haase added a comment -

          Thanks, Rick, for the updated spec. Attaching DERBY-6564.diff, DERBY-6564.stat, and DERBY-6564.zip, with the following Tools Guide changes:

          A src/tools/rtoolsoptlucene.dita
          A src/tools/rtoolsoptlucenecreate.dita
          A src/tools/rtoolsoptlucenedrop.dita
          A src/tools/rtoolsoptlucenelist.dita
          A src/tools/rtoolsoptlucenequery.dita
          A src/tools/rtoolsoptlucenesecman.dita
          A src/tools/rtoolsoptluceneupdate.dita
          M src/tools/derbytools.ditamap

          Thanks in advance for reviewing this.

          Show
          Kim Haase added a comment - Thanks, Rick, for the updated spec. Attaching DERBY-6564 .diff, DERBY-6564 .stat, and DERBY-6564 .zip, with the following Tools Guide changes: A src/tools/rtoolsoptlucene.dita A src/tools/rtoolsoptlucenecreate.dita A src/tools/rtoolsoptlucenedrop.dita A src/tools/rtoolsoptlucenelist.dita A src/tools/rtoolsoptlucenequery.dita A src/tools/rtoolsoptlucenesecman.dita A src/tools/rtoolsoptluceneupdate.dita M src/tools/derbytools.ditamap Thanks in advance for reviewing this.
          Hide
          Rick Hillegas added a comment -

          Hi Kim,

          This looks great. One small point:

          o I see that the spec still doesn't describe "score" correctly. It should just say that "score" is a float value. It could be positive, negative, or 0. So the description of "score" needs to be fixed in rtoolsoptlucene.html

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Kim, This looks great. One small point: o I see that the spec still doesn't describe "score" correctly. It should just say that "score" is a float value. It could be positive, negative, or 0. So the description of "score" needs to be fixed in rtoolsoptlucene.html Thanks, -Rick
          Hide
          Kim Haase added a comment -

          Thanks, Rick. I'm attaching DERBY-6564-2.diff and DERBY-6564-2.zip, with that correction. Hope the patch is okay now.

          Show
          Kim Haase added a comment - Thanks, Rick. I'm attaching DERBY-6564 -2.diff and DERBY-6564 -2.zip, with that correction. Hope the patch is okay now.
          Hide
          Rick Hillegas added a comment -

          Thanks, Kim. This looks great. +1

          Show
          Rick Hillegas added a comment - Thanks, Kim. This looks great. +1
          Hide
          ASF subversion and git services added a comment -

          Commit 1596291 from Kim Haase in branch 'docs/trunk'
          [ https://svn.apache.org/r1596291 ]

          DERBY-6564 Document the experimental, optional LuceneSupport tool.

          Added 7 new topics to the Tools Guide and modified the map file.

          Patches: DERBY-6564-2.diff

          Show
          ASF subversion and git services added a comment - Commit 1596291 from Kim Haase in branch 'docs/trunk' [ https://svn.apache.org/r1596291 ] DERBY-6564 Document the experimental, optional LuceneSupport tool. Added 7 new topics to the Tools Guide and modified the map file. Patches: DERBY-6564 -2.diff
          Hide
          Kim Haase added a comment -

          Thanks again, Rick!

          Committed patch DERBY-6564-2.diff to documentation trunk at revision 1596291.

          Show
          Kim Haase added a comment - Thanks again, Rick! Committed patch DERBY-6564 -2.diff to documentation trunk at revision 1596291.
          Hide
          Mike Matrigali added a comment -

          new 10.11 feature, should not be backported to 10.10.

          review of list of issues fixed in 10.11 but not in 10.10 for possible compatibility issues in upcoming release. Need to check documentation to see if
          soft vs hard upgrade behavior is documented.

          Show
          Mike Matrigali added a comment - new 10.11 feature, should not be backported to 10.10. review of list of issues fixed in 10.11 but not in 10.10 for possible compatibility issues in upcoming release. Need to check documentation to see if soft vs hard upgrade behavior is documented.
          Hide
          Kim Haase added a comment -

          Reopening to add full-upgrade info.

          Show
          Kim Haase added a comment - Reopening to add full-upgrade info.
          Hide
          Kim Haase added a comment -

          Attaching DERBY-6564-upgrade.diff and rtoolsoptlucene.html, with upgrade information in the main Lucene plugin topic:

          M src/tools/rtoolsoptlucene.dita

          Thanks in advance for reviewing this.

          Show
          Kim Haase added a comment - Attaching DERBY-6564 -upgrade.diff and rtoolsoptlucene.html, with upgrade information in the main Lucene plugin topic: M src/tools/rtoolsoptlucene.dita Thanks in advance for reviewing this.
          Hide
          Rick Hillegas added a comment -

          Thanks, Kim. Looks good to me. +1

          Show
          Rick Hillegas added a comment - Thanks, Kim. Looks good to me. +1
          Hide
          ASF subversion and git services added a comment -

          Commit 1601405 from Kim Haase in branch 'docs/trunk'
          [ https://svn.apache.org/r1601405 ]

          DERBY-6564 Document the experimental, optional LuceneSupport tool.

          Modified a Tools Guide topic to add upgrade information.

          Patches: DERBY-6564-upgrade.diff

          Show
          ASF subversion and git services added a comment - Commit 1601405 from Kim Haase in branch 'docs/trunk' [ https://svn.apache.org/r1601405 ] DERBY-6564 Document the experimental, optional LuceneSupport tool. Modified a Tools Guide topic to add upgrade information. Patches: DERBY-6564 -upgrade.diff
          Hide
          Kim Haase added a comment -

          Thanks again, Rick.

          Committed patch DERBY-6564-upgrade.diff to documentation trunk at revision 1601405.

          Show
          Kim Haase added a comment - Thanks again, Rick. Committed patch DERBY-6564 -upgrade.diff to documentation trunk at revision 1601405.
          Hide
          Rick Hillegas added a comment -

          Re-opening this issue so that we can incorporate the recent changes made to the api of the Lucene plugin. The changes are described at a high level by a 2014-06-30 comment on DERBY-590 and by the latest version of the functional spec attached to that issue.

          Show
          Rick Hillegas added a comment - Re-opening this issue so that we can incorporate the recent changes made to the api of the Lucene plugin. The changes are described at a high level by a 2014-06-30 comment on DERBY-590 and by the latest version of the functional spec attached to that issue.
          Hide
          Kim Haase added a comment -

          Thanks, Rick, for the very clear description of the work needed as well as the updated spec.

          Attaching DERBY-6564-3.diff, DERBY-6564-3.stat, and DERBY-6564-3.zip, with modifications to the following:

          M src/tools/rtoolsoptluceneupdate.dita
          M src/tools/rtoolsoptlucenequery.dita
          M src/tools/rtoolsoptlucenelist.dita
          M src/tools/rtoolsoptlucenecreate.dita

          Please let me know if I've missed something. Thanks!

          Show
          Kim Haase added a comment - Thanks, Rick, for the very clear description of the work needed as well as the updated spec. Attaching DERBY-6564 -3.diff, DERBY-6564 -3.stat, and DERBY-6564 -3.zip, with modifications to the following: M src/tools/rtoolsoptluceneupdate.dita M src/tools/rtoolsoptlucenequery.dita M src/tools/rtoolsoptlucenelist.dita M src/tools/rtoolsoptlucenecreate.dita Please let me know if I've missed something. Thanks!
          Hide
          Rick Hillegas added a comment -

          Thanks, Kim. These changes look great to me. +1

          Show
          Rick Hillegas added a comment - Thanks, Kim. These changes look great to me. +1
          Hide
          ASF subversion and git services added a comment -

          Commit 1606904 from Kim Haase in branch 'docs/trunk'
          [ https://svn.apache.org/r1606904 ]

          DERBY-6564 Document the experimental, optional LuceneSupport tool.

          Modified 4 Tools Guide topics.

          Patch: DERBY-6564-3.diff

          Show
          ASF subversion and git services added a comment - Commit 1606904 from Kim Haase in branch 'docs/trunk' [ https://svn.apache.org/r1606904 ] DERBY-6564 Document the experimental, optional LuceneSupport tool. Modified 4 Tools Guide topics. Patch: DERBY-6564 -3.diff
          Hide
          Kim Haase added a comment -

          Thanks, Rick, for the fast review.

          Committed patch DERBY-6564-3.diff to documentation trunk at revision 1606904.

          Show
          Kim Haase added a comment - Thanks, Rick, for the fast review. Committed patch DERBY-6564 -3.diff to documentation trunk at revision 1606904.

            People

            • Assignee:
              Kim Haase
              Reporter:
              Rick Hillegas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development