Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7501

Do not encode the split dimension in the index in the 1D case

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: master (7.0), 6.3
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      When there is a single dimension, the split dimension is always 0, so we do not need to encode it in the index of the BKD tree. This would be 33% memory saving for half floats, 20% for ints/floats, 11% for longs/doubles and 6% for ip addresses.

      1. LUCENE-7501.patch
        17 kB
        Michael McCandless
      2. LUCENE-7501.patch
        12 kB
        Michael McCandless
      3. LUCENE-7501.patch
        7 kB
        Adrien Grand

        Activity

        Hide
        jpountz Adrien Grand added a comment -

        Here is a patch.

        Show
        jpountz Adrien Grand added a comment - Here is a patch.
        Hide
        mikemccand Michael McCandless added a comment -

        +1, most excellent!

        Show
        mikemccand Michael McCandless added a comment - +1, most excellent!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit e6d225603aad24a7420f395ecd3dffc17fddf62c in lucene-solr's branch refs/heads/branch_6x from Adrien Grand
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e6d2256 ]

        LUCENE-7501: BKDReader should not store the split dimension explicitly in the 1D case.

        Show
        jira-bot ASF subversion and git services added a comment - Commit e6d225603aad24a7420f395ecd3dffc17fddf62c in lucene-solr's branch refs/heads/branch_6x from Adrien Grand [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e6d2256 ] LUCENE-7501 : BKDReader should not store the split dimension explicitly in the 1D case.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit e8360714fae8bac3c705297641a735ed33cf47f9 in lucene-solr's branch refs/heads/master from Adrien Grand
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e836071 ]

        LUCENE-7501: BKDReader should not store the split dimension explicitly in the 1D case.

        Show
        jira-bot ASF subversion and git services added a comment - Commit e8360714fae8bac3c705297641a735ed33cf47f9 in lucene-solr's branch refs/heads/master from Adrien Grand [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e836071 ] LUCENE-7501 : BKDReader should not store the split dimension explicitly in the 1D case.
        Hide
        mikemccand Michael McCandless added a comment -

        I think there's a small back compat bug here, that looks like corruption.

        I'll attach a quick patch, but maybe there's a cleaner way e.g. to factor out the address++ / splitDim logic.

        I also hit some silly crabs in CheckIndex.

        Show
        mikemccand Michael McCandless added a comment - I think there's a small back compat bug here, that looks like corruption. I'll attach a quick patch, but maybe there's a cleaner way e.g. to factor out the address++ / splitDim logic. I also hit some silly crabs in CheckIndex.
        Hide
        mikemccand Michael McCandless added a comment -

        I think we have to fix this for 6.3.0.

        Show
        mikemccand Michael McCandless added a comment - I think we have to fix this for 6.3.0.
        Hide
        mikemccand Michael McCandless added a comment -

        Patch.

        Show
        mikemccand Michael McCandless added a comment - Patch.
        Hide
        jpountz Adrien Grand added a comment -

        The patch looks good, thanks for looking into it. To be safe, maybe we should undo the change on the 6.3 branch and apply your patch to master/6.x?

        Show
        jpountz Adrien Grand added a comment - The patch looks good, thanks for looking into it. To be safe, maybe we should undo the change on the 6.3 branch and apply your patch to master/6.x?
        Hide
        mikemccand Michael McCandless added a comment -

        Patch w/ new test case showing the issue, and showing it's fixed w/ the patch.

        I don't think we need to pull this from 6.3.0; the change is contained, and just had a back-compat bug.

        Show
        mikemccand Michael McCandless added a comment - Patch w/ new test case showing the issue, and showing it's fixed w/ the patch. I don't think we need to pull this from 6.3.0; the change is contained, and just had a back-compat bug.
        Hide
        jpountz Adrien Grand added a comment -

        +1

        Show
        jpountz Adrien Grand added a comment - +1
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 35f3cd9bd95607983fa4704c66f61bf59cd40860 in lucene-solr's branch refs/heads/branch_6x from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=35f3cd9 ]

        LUCENE-7501: fix back-compat bug; add test

        Show
        jira-bot ASF subversion and git services added a comment - Commit 35f3cd9bd95607983fa4704c66f61bf59cd40860 in lucene-solr's branch refs/heads/branch_6x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=35f3cd9 ] LUCENE-7501 : fix back-compat bug; add test
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 69e654b3737a97fea7ffc9f57c8fad5e85f72452 in lucene-solr's branch refs/heads/master from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=69e654b ]

        LUCENE-7501: fix back-compat bug; add test

        Show
        jira-bot ASF subversion and git services added a comment - Commit 69e654b3737a97fea7ffc9f57c8fad5e85f72452 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=69e654b ] LUCENE-7501 : fix back-compat bug; add test
        Hide
        mikemccand Michael McCandless added a comment -

        Hmm I'm not sure why the commit bot doesn't show my push to branch_6_3, but I did push the fix there too!

        Show
        mikemccand Michael McCandless added a comment - Hmm I'm not sure why the commit bot doesn't show my push to branch_6_3, but I did push the fix there too!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 69e654b3737a97fea7ffc9f57c8fad5e85f72452 in lucene-solr's branch refs/heads/apiv2 from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=69e654b ]

        LUCENE-7501: fix back-compat bug; add test

        Show
        jira-bot ASF subversion and git services added a comment - Commit 69e654b3737a97fea7ffc9f57c8fad5e85f72452 in lucene-solr's branch refs/heads/apiv2 from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=69e654b ] LUCENE-7501 : fix back-compat bug; add test
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Closing after 6.3.0 release.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.

          People

          • Assignee:
            jpountz Adrien Grand
            Reporter:
            jpountz Adrien Grand
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development