Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9034

Atomic updates not work with CopyField

    Details

      Description

      Atomic updates does not work when CopyField has docValues enabled. Below is the sample schema

         <field name="single_i_dvn" multiValued="false" type="int"    indexed="true" stored="true" />
         <field name="single_d_dvn" multiValued="false" type="double" indexed="true" stored="true" />
         <field name="single_s_dvn" multiValued="false" type="string" indexed="true" stored="true" />
         <field name="copy_single_i_dvn" multiValued="false" type="int"    docValues="true" indexed="true" stored="false" useDocValuesAsStored="false" />
         <field name="copy_single_d_dvn" multiValued="false" type="double" docValues="true" indexed="true" stored="false" useDocValuesAsStored="false" />
         <field name="copy_single_s_dvn" multiValued="false" type="string" docValues="true" indexed="true" stored="false" useDocValuesAsStored="false" />
      

      Below is the exception

      Caused by: java.lang.IllegalArgumentException: DocValuesField
       "copy_single_i_dvn" appears more than once in this document 
      (only one value is allowed per field)
      
      1. SOLR-9034.patch
        9 kB
        Karthik Ramachandran
      2. SOLR-9034.patch
        9 kB
        Yonik Seeley
      3. SOLR-9034.patch
        10 kB
        Ishan Chattopadhyaya

        Activity

        Hide
        kramachandran@commvault.com Karthik Ramachandran added a comment -

        Can someone review the change? It would good if this fix makes it to 5.5.1

        Show
        kramachandran@commvault.com Karthik Ramachandran added a comment - Can someone review the change? It would good if this fix makes it to 5.5.1
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Thanks for the patch, I am reviewing it.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Thanks for the patch, I am reviewing it.
        Hide
        yseeley@gmail.com Yonik Seeley added a comment -

        We should also make sure that we have tests for both retrieval cases (transaction log and index).

        Show
        yseeley@gmail.com Yonik Seeley added a comment - We should also make sure that we have tests for both retrieval cases (transaction log and index).
        Hide
        yseeley@gmail.com Yonik Seeley added a comment -

        Updated patch... the test modification should improve coverage for the retrieval-from-tlog case.

        Show
        yseeley@gmail.com Yonik Seeley added a comment - Updated patch... the test modification should improve coverage for the retrieval-from-tlog case.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Updating the patch to move the logic of filtering the list of nonstored DV fields that are copy targets to the SolrIndexSearcher so that the list is computed once and used again and again. The logic for filtering, as in Karthik's original patch, seems like a good idea in order to solve the issue, since such a filtering happens for stored fields that are copy field targets. This patch retains the tests from Yonik's previous patch.
        Karthik Ramachandran, Yonik Seeley, please review.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Updating the patch to move the logic of filtering the list of nonstored DV fields that are copy targets to the SolrIndexSearcher so that the list is computed once and used again and again. The logic for filtering, as in Karthik's original patch, seems like a good idea in order to solve the issue, since such a filtering happens for stored fields that are copy field targets. This patch retains the tests from Yonik's previous patch. Karthik Ramachandran , Yonik Seeley , please review.
        Hide
        kramachandran@commvault.com Karthik Ramachandran added a comment -

        Changes looks good.

        Show
        kramachandran@commvault.com Karthik Ramachandran added a comment - Changes looks good.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit c897917c718eef75d66c5d0006f409d5c95260c7 in lucene-solr's branch refs/heads/master from Yonik Seeley
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c897917 ]

        SOLR-9034: fix atomic updates for copyField w/ docValues

        Show
        jira-bot ASF subversion and git services added a comment - Commit c897917c718eef75d66c5d0006f409d5c95260c7 in lucene-solr's branch refs/heads/master from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c897917 ] SOLR-9034 : fix atomic updates for copyField w/ docValues
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 21aea6f606f81b1b4c45fa41501f33744f2b887a in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=21aea6f ]

        SOLR-9034: fix atomic updates for copyField w/ docValues

        Show
        jira-bot ASF subversion and git services added a comment - Commit 21aea6f606f81b1b4c45fa41501f33744f2b887a in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=21aea6f ] SOLR-9034 : fix atomic updates for copyField w/ docValues
        Hide
        yseeley@gmail.com Yonik Seeley added a comment -

        Committed. Thanks!

        Show
        yseeley@gmail.com Yonik Seeley added a comment - Committed. Thanks!
        Hide
        przemosz Przemysław Szeremiota added a comment - - edited

        Yonik,

        I was fighting SOLR 5.5 useDocValuesAsStored/copyField issue on our company's SOLR installation, and sufficient fix seems to be simple:

        solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
          -       searcher.decorateDocValueFields(sid, docid, searcher.getNonStoredDVs(false));
          +       searcher.decorateDocValueFields(sid, docid, searcher.getNonStoredDVs(true));
        

        getNonStoredDVs(false) returns all non-stored docValues fields, getNonStoredDVs(true) returns only non-stored docValues fields used as stored, either explicitly or implicitly (in schema 1.6). Doesn't masking "implicitly use all docvalues as stored, configured or not" behavior with copyField target detection defeats whole purpose of choosing docValues/stored behavior?

        Show
        przemosz Przemysław Szeremiota added a comment - - edited Yonik, I was fighting SOLR 5.5 useDocValuesAsStored/copyField issue on our company's SOLR installation, and sufficient fix seems to be simple: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java - searcher.decorateDocValueFields(sid, docid, searcher.getNonStoredDVs( false )); + searcher.decorateDocValueFields(sid, docid, searcher.getNonStoredDVs( true )); getNonStoredDVs(false) returns all non-stored docValues fields, getNonStoredDVs(true) returns only non-stored docValues fields used as stored, either explicitly or implicitly (in schema 1.6). Doesn't masking "implicitly use all docvalues as stored, configured or not" behavior with copyField target detection defeats whole purpose of choosing docValues/stored behavior?
        Hide
        steve_rowe Steve Rowe added a comment -

        Reopening to backport to 6.0.1.

        Show
        steve_rowe Steve Rowe added a comment - Reopening to backport to 6.0.1.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit be59da0b42df01a2e4b6c6fa8d8d6bf50fb3d690 in lucene-solr's branch refs/heads/branch_6_0 from Yonik Seeley
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=be59da0 ]

        SOLR-9034: fix atomic updates for copyField w/ docValues

        Show
        jira-bot ASF subversion and git services added a comment - Commit be59da0b42df01a2e4b6c6fa8d8d6bf50fb3d690 in lucene-solr's branch refs/heads/branch_6_0 from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=be59da0 ] SOLR-9034 : fix atomic updates for copyField w/ docValues
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ed028a75dde0cc2cfe577027f60a5634aa5e7c13 in lucene-solr's branch refs/heads/branch_6_0 from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ed028a7 ]

        SOLR-9034: branch_6_0: move CHANGES entry from under 5.5.1 to under 6.0.1

        Show
        jira-bot ASF subversion and git services added a comment - Commit ed028a75dde0cc2cfe577027f60a5634aa5e7c13 in lucene-solr's branch refs/heads/branch_6_0 from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ed028a7 ] SOLR-9034 : branch_6_0: move CHANGES entry from under 5.5.1 to under 6.0.1
        Hide
        dragonsinth Scott Blum added a comment -

        Did this make 5.5.1?

        Show
        dragonsinth Scott Blum added a comment - Did this make 5.5.1?
        Hide
        steve_rowe Steve Rowe added a comment -

        Bulk close issues included in the 6.0.1 release.

        Show
        steve_rowe Steve Rowe added a comment - Bulk close issues included in the 6.0.1 release.
        Hide
        dragonsinth Scott Blum added a comment -

        Should probably backport to 5.5.2, 5.6

        Show
        dragonsinth Scott Blum added a comment - Should probably backport to 5.5.2, 5.6
        Hide
        steve_rowe Steve Rowe added a comment -

        Reopening to backport to 5.6 and 5.5.2.

        Show
        steve_rowe Steve Rowe added a comment - Reopening to backport to 5.6 and 5.5.2.
        Hide
        steve_rowe Steve Rowe added a comment -

        Here are the 5.6 and 5.5.2 commits, since the ASF bot doesn't seem to be working:


        branch_5x: SOLR-9034: fix atomic updates for copyField w/ docValues
        Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5f91aa95


        branch_5_5: SOLR-9034: fix atomic updates for copyField w/ docValues
        Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/7777599d


        branch_5_5: SOLR-9034: Add 5.5.2 CHANGES entry
        Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6f06e56c


        Show
        steve_rowe Steve Rowe added a comment - Here are the 5.6 and 5.5.2 commits, since the ASF bot doesn't seem to be working: branch_5x: SOLR-9034 : fix atomic updates for copyField w/ docValues Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5f91aa95 branch_5_5: SOLR-9034 : fix atomic updates for copyField w/ docValues Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/7777599d branch_5_5: SOLR-9034 : Add 5.5.2 CHANGES entry Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6f06e56c
        Hide
        steve_rowe Steve Rowe added a comment -

        My Jenkins found a reproducing TestNRTOpen.testSharedCores() failure on branch_5_5, and git bisect points the finger at this issue - the seed doesn't reproduce for me on master, branch_6x, branch_6_1 or branch_6_0 though:

        Checking out Revision c6b9ac065571718e7e92174fa7e2a927583012fa (refs/remotes/origin/branch_5_5)
        [...]
           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestNRTOpen -Dtests.method=testSharedCores -Dtests.seed=7C31F56A2C49EB56 -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar-LB -Dtests.timezone=Asia/Magadan -Dtests.asserts=true -Dtests.file.encoding=UTF-8
           [junit4] FAILURE 0.21s J5  | TestNRTOpen.testSharedCores <<<
           [junit4]    > Throwable #1: java.lang.AssertionError
           [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7C31F56A2C49EB56:9A32BF25A4BFC659]:0)
           [junit4]    > 	at org.apache.solr.core.TestNRTOpen.testSharedCores(TestNRTOpen.java:117)
           [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
        
        Show
        steve_rowe Steve Rowe added a comment - My Jenkins found a reproducing TestNRTOpen.testSharedCores() failure on branch_5_5, and git bisect points the finger at this issue - the seed doesn't reproduce for me on master, branch_6x, branch_6_1 or branch_6_0 though: Checking out Revision c6b9ac065571718e7e92174fa7e2a927583012fa (refs/remotes/origin/branch_5_5) [...] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestNRTOpen -Dtests.method=testSharedCores -Dtests.seed=7C31F56A2C49EB56 -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar-LB -Dtests.timezone=Asia/Magadan -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] FAILURE 0.21s J5 | TestNRTOpen.testSharedCores <<< [junit4] > Throwable #1: java.lang.AssertionError [junit4] > at __randomizedtesting.SeedInfo.seed([7C31F56A2C49EB56:9A32BF25A4BFC659]:0) [junit4] > at org.apache.solr.core.TestNRTOpen.testSharedCores(TestNRTOpen.java:117) [junit4] > at java.lang.Thread.run(Thread.java:745)
        Hide
        steve_rowe Steve Rowe added a comment -

        Another reproducing branch_5_5 seed for TestNRTOpen.testSharedCores() http://jenkins.thetaphi.de/job/Lucene-Solr-5.5-Linux/310/consoleText:

        Checking out Revision c6b9ac065571718e7e92174fa7e2a927583012fa (refs/remotes/origin/branch_5_5)
        [...]
           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestNRTOpen -Dtests.method=testSharedCores -Dtests.seed=A2B2DA22BF73E3AF -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-HN -Dtests.timezone=Libya -Dtests.asserts=true -Dtests.file.encoding=UTF-8
           [junit4] FAILURE 0.49s J1 | TestNRTOpen.testSharedCores <<<
           [junit4]    > Throwable #1: java.lang.AssertionError: expected:<3> but was:<1>
           [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A2B2DA22BF73E3AF:44B1906D3785CEA0]:0)
           [junit4]    > 	at org.apache.solr.core.TestNRTOpen.testSharedCores(TestNRTOpen.java:116)
        

        But this seed continues to fail even when I rollback to the commit prior to this issue's branch_5_5 commit. Also, when I run the test on my box the failure point and cause are different - this is on branch_5_5 at the SOLR-9034 commit point on branch_5_5 (7777599):

           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestNRTOpen -Dtests.method=testSharedCores -Dtests.seed=A2B2DA22BF73E3AF -Dtests.multiplier=3 -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es-HN -Dtests.timezone=Libya -Dtests.asserts=true -Dtests.file.encoding=UTF-8
           [junit4] FAILURE 0.27s | TestNRTOpen.testSharedCores <<<
           [junit4]    > Throwable #1: java.lang.AssertionError: expected:<[org.apache.lucene.index.SegmentCoreReaders@6cda66e6, org.apache.lucene.index.SegmentCoreReaders@37203df2, org.apache.lucene.index.SegmentCoreReaders@49f95cf6]> but was:<[org.apache.lucene.index.SegmentCoreReaders@345238b3]>
           [junit4]    >        at __randomizedtesting.SeedInfo.seed([A2B2DA22BF73E3AF:44B1906D3785CEA0]:0)
           [junit4]    >        at org.apache.solr.core.TestNRTOpen.testSharedCores(TestNRTOpen.java:124)
        

        So it looks like the other seed succeeds earlier than the SOLR-9034 commit for indirect reasons. I'll poke around some more.

        Show
        steve_rowe Steve Rowe added a comment - Another reproducing branch_5_5 seed for TestNRTOpen.testSharedCores() http://jenkins.thetaphi.de/job/Lucene-Solr-5.5-Linux/310/consoleText : Checking out Revision c6b9ac065571718e7e92174fa7e2a927583012fa (refs/remotes/origin/branch_5_5) [...] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestNRTOpen -Dtests.method=testSharedCores -Dtests.seed=A2B2DA22BF73E3AF -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-HN -Dtests.timezone=Libya -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] FAILURE 0.49s J1 | TestNRTOpen.testSharedCores <<< [junit4] > Throwable #1: java.lang.AssertionError: expected:<3> but was:<1> [junit4] > at __randomizedtesting.SeedInfo.seed([A2B2DA22BF73E3AF:44B1906D3785CEA0]:0) [junit4] > at org.apache.solr.core.TestNRTOpen.testSharedCores(TestNRTOpen.java:116) But this seed continues to fail even when I rollback to the commit prior to this issue's branch_5_5 commit. Also, when I run the test on my box the failure point and cause are different - this is on branch_5_5 at the SOLR-9034 commit point on branch_5_5 (7777599): [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestNRTOpen -Dtests.method=testSharedCores -Dtests.seed=A2B2DA22BF73E3AF -Dtests.multiplier=3 -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es-HN -Dtests.timezone=Libya -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] FAILURE 0.27s | TestNRTOpen.testSharedCores <<< [junit4] > Throwable #1: java.lang.AssertionError: expected:<[org.apache.lucene.index.SegmentCoreReaders@6cda66e6, org.apache.lucene.index.SegmentCoreReaders@37203df2, org.apache.lucene.index.SegmentCoreReaders@49f95cf6]> but was:<[org.apache.lucene.index.SegmentCoreReaders@345238b3]> [junit4] > at __randomizedtesting.SeedInfo.seed([A2B2DA22BF73E3AF:44B1906D3785CEA0]:0) [junit4] > at org.apache.solr.core.TestNRTOpen.testSharedCores(TestNRTOpen.java:124) So it looks like the other seed succeeds earlier than the SOLR-9034 commit for indirect reasons. I'll poke around some more.
        Hide
        steve_rowe Steve Rowe added a comment -

        I've opened SOLR-9229 for the TestNRTOpen.testSharedCores() failures, and since they don't seem to be directly related to this issue - more likely SOLR-8621 is the cause - I'm resolving this issue.

        Show
        steve_rowe Steve Rowe added a comment - I've opened SOLR-9229 for the TestNRTOpen.testSharedCores() failures, and since they don't seem to be directly related to this issue - more likely SOLR-8621 is the cause - I'm resolving this issue.
        Hide
        steve_rowe Steve Rowe added a comment -

        Bulk close issues released with 5.5.2.

        Show
        steve_rowe Steve Rowe added a comment - Bulk close issues released with 5.5.2.

          People

          • Assignee:
            yseeley@gmail.com Yonik Seeley
            Reporter:
            kramachandran@commvault.com Karthik Ramachandran
          • Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development