Lucene - Core
  1. Lucene - Core
  2. LUCENE-5619

TestBackwardsCompatibility needs updatable docvalues

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      We don't test this at all in TestBackCompat. this is scary!

      1. dvupdates.48.zip
        10 kB
        Shai Erera
      2. LUCENE-5619.patch
        5 kB
        Shai Erera

        Activity

        Hide
        Robert Muir added a comment -

        I will look into this: first we should generate some updates for the versions of lucene that support this.

        Second we should try to apply updates against old indexes. I am not sure about the rules here: is it ok to apply updates to e.g. a 3.x or 4.0 index? I guess we will know soon...

        Show
        Robert Muir added a comment - I will look into this: first we should generate some updates for the versions of lucene that support this. Second we should try to apply updates against old indexes. I am not sure about the rules here: is it ok to apply updates to e.g. a 3.x or 4.0 index? I guess we will know soon...
        Hide
        Shai Erera added a comment -

        I am looking into this since it's important to have it in place before the work on LUCENE-5618 (and LUCENE-5636).

        I am not sure about the rules here: is it ok to apply updates to e.g. a 3.x or 4.0 index?

        No, updating those indexes is not supported (we even suppress those codecs in tests), since those codecs did not take the segmentSuffix into account. We've decided that in order to use updatable DocValues, you need to index with 4.6+, or re-create the index if it was created with earlier versions. More so, old formats' consumers aren't shipped w/ Lucene anyway.

        I'll look into adding those indexes to TestBackCompat – recreate them w/ few numeric and binary doc-values and then try to update them with newer code.

        Show
        Shai Erera added a comment - I am looking into this since it's important to have it in place before the work on LUCENE-5618 (and LUCENE-5636 ). I am not sure about the rules here: is it ok to apply updates to e.g. a 3.x or 4.0 index? No, updating those indexes is not supported (we even suppress those codecs in tests), since those codecs did not take the segmentSuffix into account. We've decided that in order to use updatable DocValues, you need to index with 4.6+, or re-create the index if it was created with earlier versions. More so, old formats' consumers aren't shipped w/ Lucene anyway. I'll look into adding those indexes to TestBackCompat – recreate them w/ few numeric and binary doc-values and then try to update them with newer code.
        Hide
        Shai Erera added a comment -

        Patch adds a test to TestBackCompat: one commented-out test builds an index with DocValues updates and one test ensures that we can update old indexes.

        I also attached such "old" index created from the lucene_solr_4_8 branch. To test it, simply drop it under lucene/core/src/test/o.a.l/index/.

        I am not sure if I should create similar indexes for 4.6 (when NDV capability was added), or it's enough to have only 4.8 for now?

        Show
        Shai Erera added a comment - Patch adds a test to TestBackCompat: one commented-out test builds an index with DocValues updates and one test ensures that we can update old indexes. I also attached such "old" index created from the lucene_solr_4_8 branch. To test it, simply drop it under lucene/core/src/test/o.a.l/index/. I am not sure if I should create similar indexes for 4.6 (when NDV capability was added), or it's enough to have only 4.8 for now?
        Hide
        Shai Erera added a comment -

        If there are no objections, I will commit it later today...

        Show
        Shai Erera added a comment - If there are no objections, I will commit it later today...
        Hide
        ASF subversion and git services added a comment -

        Commit 1594561 from Shai Erera in branch 'dev/trunk'
        [ https://svn.apache.org/r1594561 ]

        LUCENE-5619: add back-compat index+test for doc-values updates

        Show
        ASF subversion and git services added a comment - Commit 1594561 from Shai Erera in branch 'dev/trunk' [ https://svn.apache.org/r1594561 ] LUCENE-5619 : add back-compat index+test for doc-values updates
        Hide
        ASF subversion and git services added a comment -

        Commit 1594566 from Shai Erera in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1594566 ]

        LUCENE-5619: add back-compat index+test for doc-values updates

        Show
        ASF subversion and git services added a comment - Commit 1594566 from Shai Erera in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1594566 ] LUCENE-5619 : add back-compat index+test for doc-values updates
        Hide
        ASF subversion and git services added a comment -

        Commit 1594568 from Shai Erera in branch 'dev/branches/lucene_solr_4_8'
        [ https://svn.apache.org/r1594568 ]

        LUCENE-5619: add back-compat index generation code for doc-values updates

        Show
        ASF subversion and git services added a comment - Commit 1594568 from Shai Erera in branch 'dev/branches/lucene_solr_4_8' [ https://svn.apache.org/r1594568 ] LUCENE-5619 : add back-compat index generation code for doc-values updates
        Hide
        Shai Erera added a comment -

        Committed to trunk and 4x the index generation code as well as a testcase. To 4.8.x I committed only the index generation code, commented out.

        Show
        Shai Erera added a comment - Committed to trunk and 4x the index generation code as well as a testcase. To 4.8.x I committed only the index generation code, commented out.

          People

          • Assignee:
            Shai Erera
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development