Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10046

Create UninvertDocValuesMergePolicy

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Create a merge policy that can detect schema changes and use UninvertingReader to uninvert fields and write docvalues into merged segments when a field has docvalues enabled.

      The current behavior is to write null values in the merged segment which can lead to data integrity problems when sorting or faceting pending a full reindex.

      With this patch it would still be recommended to reindex when adding docvalues for performance reasons, as it not guarenteed all segments will be merged with docvalues turned on.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kelaban opened a pull request:

          https://github.com/apache/lucene-solr/pull/144

          SOLR-10046 - Add UninvertDocValuesMergePolicyFactory and lucene/OneMe…

          …rgeWrappingMergePolicy

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/kelaban/lucene-solr jira/master/SOLR-10046/create-uninvert-docvalues-merge-policy

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/lucene-solr/pull/144.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #144


          commit bf8877fcb909422de5427ee9a7b791570a92ae22
          Author: Keith Laban <klaban1@bloomberg.net>
          Date: 2017-01-25T20:43:23Z

          SOLR-10046 - Add UninvertDocValuesMergePolicyFactory and lucene/OneMergeWrappingMergePolicy


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kelaban opened a pull request: https://github.com/apache/lucene-solr/pull/144 SOLR-10046 - Add UninvertDocValuesMergePolicyFactory and lucene/OneMe… …rgeWrappingMergePolicy You can merge this pull request into a Git repository by running: $ git pull https://github.com/kelaban/lucene-solr jira/master/ SOLR-10046 /create-uninvert-docvalues-merge-policy Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/144.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #144 commit bf8877fcb909422de5427ee9a7b791570a92ae22 Author: Keith Laban <klaban1@bloomberg.net> Date: 2017-01-25T20:43:23Z SOLR-10046 - Add UninvertDocValuesMergePolicyFactory and lucene/OneMergeWrappingMergePolicy
          Hide
          cpoerschke Christine Poerschke added a comment -

          Hi Keith,

          Thank you for opening this ticket with a pull request.

          Thanks also for reviewing and including my suggested changes for the lucene part, the solr suggestions are separate just for clarity.

          (Split out and linked to this ticket for visibility, LUCENE-7688 and LUCENE-7689 re: the OneMergeWrappingMergePolicy location (lucene vs. solr) and making ant documentation-lint (part of ant recommit) pass with it.)

          Show
          cpoerschke Christine Poerschke added a comment - Hi Keith, Thank you for opening this ticket with a pull request. Thanks also for reviewing and including my suggested changes for the lucene part, the solr suggestions are separate just for clarity. (Split out and linked to this ticket for visibility, LUCENE-7688 and LUCENE-7689 re: the OneMergeWrappingMergePolicy location (lucene vs. solr) and making ant documentation-lint (part of ant recommit ) pass with it.)
          Hide
          k317h Keith Laban added a comment -

          Christine, I merged your second PR, not sure that was required or not. Thanks for your suggestions, it's looking good.

          Show
          k317h Keith Laban added a comment - Christine, I merged your second PR, not sure that was required or not. Thanks for your suggestions, it's looking good.
          Hide
          cpoerschke Christine Poerschke added a comment -

          Hi Keith,

          I've committed the addition of the OneMergeWrappingMergePolicy class in LUCENE-7688 yesterday and so then here the next steps would probably be to:

          • merge master into the create-uninvert-docvalues-merge-policy working branch
          • add the missing javadocs (TODOs)
          • check that ant precommit passes (LUCENE-7689 had turned out to be a missing javadocs issue)

          I won't have bandwidth for the above steps this week i.e. feel free to go ahead if you have time. Thanks.

          Show
          cpoerschke Christine Poerschke added a comment - Hi Keith, I've committed the addition of the OneMergeWrappingMergePolicy class in LUCENE-7688 yesterday and so then here the next steps would probably be to: merge master into the create-uninvert-docvalues-merge-policy working branch add the missing javadocs (TODOs) check that ant precommit passes ( LUCENE-7689 had turned out to be a missing javadocs issue) I won't have bandwidth for the above steps this week i.e. feel free to go ahead if you have time. Thanks.
          Hide
          k317h Keith Laban added a comment -

          Hi Christine, I was able to do the above.

          • I created a new commit on top of master to clean up the working branch
          • Added javadocs and removed TODOs
          • ant precommit passes
          Show
          k317h Keith Laban added a comment - Hi Christine, I was able to do the above. I created a new commit on top of master to clean up the working branch Added javadocs and removed TODOs ant precommit passes
          Hide
          cpoerschke Christine Poerschke added a comment -

          Thanks Keith! Just sent you a small fix pull request to do with the new skipIntegrityChecks property ...

          ... other than that I have no further comments or observations here except that I'd appreciate extra pair(s) of eyes before proceeding to commit the change towards the end of this or the beginning of this week.

          Show
          cpoerschke Christine Poerschke added a comment - Thanks Keith! Just sent you a small fix pull request to do with the new skipIntegrityChecks property ... ... other than that I have no further comments or observations here except that I'd appreciate extra pair(s) of eyes before proceeding to commit the change towards the end of this or the beginning of this week.
          Hide
          k317h Keith Laban added a comment -

          Thanks Christine, I missed this last comment. I merged your pull request

          Show
          k317h Keith Laban added a comment - Thanks Christine, I missed this last comment. I merged your pull request
          Hide
          cpoerschke Christine Poerschke added a comment -

          Just tried to apply the change to the current master branch and getting this (compile) error:

          UninvertDocValuesMergePolicyFactory.UninvertingFilterCodecReader is not abstract and does not override abstract method getCoreCacheHelper() in LeafReader
          
          Show
          cpoerschke Christine Poerschke added a comment - Just tried to apply the change to the current master branch and getting this (compile) error: UninvertDocValuesMergePolicyFactory.UninvertingFilterCodecReader is not abstract and does not override abstract method getCoreCacheHelper() in LeafReader
          Hide
          k317h Keith Laban added a comment -

          I rebased with master and added in the new get*CacheHelper methods which were added in LUCENE-7410 although I think that the delegations I added should be in the abstract FilterCodecReader instead

          Show
          k317h Keith Laban added a comment - I rebased with master and added in the new get*CacheHelper methods which were added in LUCENE-7410 although I think that the delegations I added should be in the abstract FilterCodecReader instead
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9d56f136505098ea5538c5d6eaaf60536848feb9 in lucene-solr's branch refs/heads/master from Christine Poerschke
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9d56f13 ]

          SOLR-10046: Add UninvertDocValuesMergePolicyFactory class. (Keith Laban, Christine Poerschke)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9d56f136505098ea5538c5d6eaaf60536848feb9 in lucene-solr's branch refs/heads/master from Christine Poerschke [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9d56f13 ] SOLR-10046 : Add UninvertDocValuesMergePolicyFactory class. (Keith Laban, Christine Poerschke)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 93111dab74546df85cb86d10e0b40fa2f7a109e4 in lucene-solr's branch refs/heads/jira/SOLR-10046-branch_6x from Christine Poerschke
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=93111da ]

          SOLR-10046: Add UninvertDocValuesMergePolicyFactory class. (Keith Laban, Christine Poerschke)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 93111dab74546df85cb86d10e0b40fa2f7a109e4 in lucene-solr's branch refs/heads/jira/ SOLR-10046 -branch_6x from Christine Poerschke [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=93111da ] SOLR-10046 : Add UninvertDocValuesMergePolicyFactory class. (Keith Laban, Christine Poerschke)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f743b46191eaead9e2f5014a873c57795f5e0f94 in lucene-solr's branch refs/heads/jira/SOLR-10046-branch_6x from Christine Poerschke
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f743b46 ]

          SOLR-10046: master-to-branch_6x backport adjustments (src only, test adjustments still needed)

          Show
          jira-bot ASF subversion and git services added a comment - Commit f743b46191eaead9e2f5014a873c57795f5e0f94 in lucene-solr's branch refs/heads/jira/ SOLR-10046 -branch_6x from Christine Poerschke [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f743b46 ] SOLR-10046 : master-to-branch_6x backport adjustments (src only, test adjustments still needed)
          Hide
          cpoerschke Christine Poerschke added a comment -

          Thanks Keith! Above commit is for master branch, for the backporting to branch_6x some adjustments appear to be needed, I've made a start on jira/SOLR-10046-branch_6x working branch w.r.t. src changes but test changes still remain to be done; I won't have more time for it today, hence sharing the working branch in case you might have time today or this week.

          Show
          cpoerschke Christine Poerschke added a comment - Thanks Keith! Above commit is for master branch, for the backporting to branch_6x some adjustments appear to be needed, I've made a start on jira/SOLR-10046-branch_6x working branch w.r.t. src changes but test changes still remain to be done; I won't have more time for it today, hence sharing the working branch in case you might have time today or this week.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ffaa2345b40cff5287fcf8edaf9059cf3a8d1892 in lucene-solr's branch refs/heads/master from Christine Poerschke
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ffaa234 ]

          SOLR-10046: move from 6.5.0 to 6.6.0 CHANGES.txt (backport yet to be completed)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ffaa2345b40cff5287fcf8edaf9059cf3a8d1892 in lucene-solr's branch refs/heads/master from Christine Poerschke [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ffaa234 ] SOLR-10046 : move from 6.5.0 to 6.6.0 CHANGES.txt (backport yet to be completed)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 09373aaa0875b8ae2bb795d5dfafbdb1450546cc in lucene-solr's branch refs/heads/branch_6x from Christine Poerschke
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=09373aa ]

          SOLR-10046: remove CHANGES.txt entry

          (Reverses unintentional add alongside SOLR-10085 and SOLR-10254 CHANGES.txt update.)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 09373aaa0875b8ae2bb795d5dfafbdb1450546cc in lucene-solr's branch refs/heads/branch_6x from Christine Poerschke [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=09373aa ] SOLR-10046 : remove CHANGES.txt entry (Reverses unintentional add alongside SOLR-10085 and SOLR-10254 CHANGES.txt update.)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 4d3e94befcb5ea361ceff1fcff1bdc3e6166fdf1 in lucene-solr's branch refs/heads/branch_6_5 from Christine Poerschke
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4d3e94b ]

          SOLR-10046: remove CHANGES.txt entry

          (Reverses unintentional add alongside SOLR-10085 and SOLR-10254 CHANGES.txt update.)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 4d3e94befcb5ea361ceff1fcff1bdc3e6166fdf1 in lucene-solr's branch refs/heads/branch_6_5 from Christine Poerschke [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4d3e94b ] SOLR-10046 : remove CHANGES.txt entry (Reverses unintentional add alongside SOLR-10085 and SOLR-10254 CHANGES.txt update.)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 90b3ef18dee4d7f583d08047da3bd95d49d859cd in lucene-solr's branch refs/heads/master from Christine Poerschke
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=90b3ef1 ]

          SOLR-10046: move from 6.6.0 to 7.0.0 CHANGES.txt (backport yet to be completed)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 90b3ef18dee4d7f583d08047da3bd95d49d859cd in lucene-solr's branch refs/heads/master from Christine Poerschke [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=90b3ef1 ] SOLR-10046 : move from 6.6.0 to 7.0.0 CHANGES.txt (backport yet to be completed)
          Hide
          cpoerschke Christine Poerschke added a comment -

          Thanks Keith!

          Show
          cpoerschke Christine Poerschke added a comment - Thanks Keith!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d0a99af7efe81f2090663f23823e8f4104b837bf in lucene-solr's branch refs/heads/master from Cassandra Targett
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d0a99af ]

          Ref Guide: Doc updates for SOLR-10046, SOLR-10929, SOLR-11021

          Show
          jira-bot ASF subversion and git services added a comment - Commit d0a99af7efe81f2090663f23823e8f4104b837bf in lucene-solr's branch refs/heads/master from Cassandra Targett [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d0a99af ] Ref Guide: Doc updates for SOLR-10046 , SOLR-10929 , SOLR-11021
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 4174d0b6ff7ad7a3690a2696e098caff0dc533ab in lucene-solr's branch refs/heads/branch_7x from Cassandra Targett
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4174d0b ]

          Ref Guide: Doc updates for SOLR-10046, SOLR-10929, SOLR-11021

          Show
          jira-bot ASF subversion and git services added a comment - Commit 4174d0b6ff7ad7a3690a2696e098caff0dc533ab in lucene-solr's branch refs/heads/branch_7x from Cassandra Targett [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4174d0b ] Ref Guide: Doc updates for SOLR-10046 , SOLR-10929 , SOLR-11021
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a0e53456bcffd610163d8eb4985e3cb65222273e in lucene-solr's branch refs/heads/branch_7_0 from Cassandra Targett
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a0e5345 ]

          Ref Guide: Doc updates for SOLR-10046, SOLR-10929, SOLR-11021

          Show
          jira-bot ASF subversion and git services added a comment - Commit a0e53456bcffd610163d8eb4985e3cb65222273e in lucene-solr's branch refs/heads/branch_7_0 from Cassandra Targett [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a0e5345 ] Ref Guide: Doc updates for SOLR-10046 , SOLR-10929 , SOLR-11021

            People

            • Assignee:
              cpoerschke Christine Poerschke
              Reporter:
              k317h Keith Laban
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development