Solr
  1. Solr
  2. SOLR-6924

TestSolrConfigHandlerCloud fails frequently.

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      I see this fail all the time. Usually something like:

      java.lang.AssertionError: Could not get expected value P val for path [response, params, y, p] full output {

      1. SOLR-6924.patch
        29 kB
        Noble Paul
      2. SOLR-6924.patch
        5 kB
        Noble Paul

        Issue Links

          Activity

          Hide
          Mark Miller added a comment -
          java.lang.AssertionError: Could not get expected value  null for path [response, params, y, c] full output {
            "responseHeader":{
              "status":0,
              "QTime":0},
            "response":{
              "znodeVersion":1,
              "params":{
                "x":{
                  "a":"A val",
                  "b":"B val",
                  "":{"v":0}},
                "y":{
                  "c":"CY val",
                  "b":"BY val",
                  "":{"v":0}}}}}
          	at __randomizedtesting.SeedInfo.seed([D765D9019AAF2D1E:56835719EDF04D22]:0)
          	at org.junit.Assert.fail(Assert.java:93)
          	at org.junit.Assert.assertTrue(Assert.java:43)
          	at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:259)
          	at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqParams(TestSolrConfigHandlerCloud.java:270)
          	at org.apache.solr.handler.TestSolrConfigHandlerCloud.doTest(TestSolrConfigHandlerCloud.java:70)
          	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
          
          Show
          Mark Miller added a comment - java.lang.AssertionError: Could not get expected value null for path [response, params, y, c] full output { "responseHeader":{ "status":0, "QTime":0}, "response":{ "znodeVersion":1, "params":{ "x":{ "a":"A val", "b":"B val", "":{"v":0}}, "y":{ "c":"CY val", "b":"BY val", "":{"v":0}}}}} at __randomizedtesting.SeedInfo.seed([D765D9019AAF2D1E:56835719EDF04D22]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:259) at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqParams(TestSolrConfigHandlerCloud.java:270) at org.apache.solr.handler.TestSolrConfigHandlerCloud.doTest(TestSolrConfigHandlerCloud.java:70) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
          Hide
          ASF subversion and git services added a comment -

          Commit 1655736 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1655736 ]

          SOLR-6924: Add BadApple to test.

          Show
          ASF subversion and git services added a comment - Commit 1655736 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1655736 ] SOLR-6924 : Add BadApple to test.
          Hide
          ASF subversion and git services added a comment -

          Commit 1655737 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1655737 ]

          SOLR-6924: Add BadApple to test.

          Show
          ASF subversion and git services added a comment - Commit 1655737 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1655737 ] SOLR-6924 : Add BadApple to test.
          Hide
          ASF subversion and git services added a comment -

          Commit 1657761 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1657761 ]

          SOLR-6924: Splitting the testcase because more SolrConfigHandler features and testcases are coming up

          Show
          ASF subversion and git services added a comment - Commit 1657761 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1657761 ] SOLR-6924 : Splitting the testcase because more SolrConfigHandler features and testcases are coming up
          Hide
          ASF subversion and git services added a comment -

          Commit 1657762 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1657762 ]

          SOLR-6924: Splitting the testcase because more SolrConfigHandler features and testcases are coming up

          Show
          ASF subversion and git services added a comment - Commit 1657762 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1657762 ] SOLR-6924 : Splitting the testcase because more SolrConfigHandler features and testcases are coming up
          Hide
          Mark Miller added a comment -
          Could not get expected value  'P val' for path 'response/params/y/p' full output: {   "responseHeader":{     "status":0,     "QTime":0},   "response":{     "znodeVersion":0,     "params":{"x":{         "a":"A val",         "b":"B val",         "":{"v":0}}}}}
          
          java.lang.AssertionError: Could not get expected value  'P val' for path 'response/params/y/p' full output: {
            "responseHeader":{
              "status":0,
              "QTime":0},
            "response":{
              "znodeVersion":0,
              "params":{"x":{
                  "a":"A val",
                  "b":"B val",
                  "":{"v":0}}}}}
          	at __randomizedtesting.SeedInfo.seed([A4ACF6659668D699:2CF8C9BF3894BB61]:0)
          	at org.junit.Assert.fail(Assert.java:93)
          	at org.junit.Assert.assertTrue(Assert.java:43)
          	at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:399)
          
          Show
          Mark Miller added a comment - Could not get expected value 'P val' for path 'response/params/y/p' full output: { "responseHeader":{ "status":0, "QTime":0}, "response":{ "znodeVersion":0, "params":{"x":{ "a":"A val", "b":"B val", "":{"v":0}}}}} java.lang.AssertionError: Could not get expected value 'P val' for path 'response/params/y/p' full output: { "responseHeader":{ "status":0, "QTime":0}, "response":{ "znodeVersion":0, "params":{"x":{ "a":"A val", "b":"B val", "":{"v":0}}}}} at __randomizedtesting.SeedInfo.seed([A4ACF6659668D699:2CF8C9BF3894BB61]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:399)
          Hide
          Shalin Shekhar Mangar added a comment -

          Another failure from my local jenkins:

           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrConfigHandlerCloud -Dtests.method=test -Dtests.seed=59310820B646978D -Dtests.slow=true -Dtests.locale=th_TH -Dtests.timezone=Africa/Sao_Tome -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
             [junit4] FAILURE 37.5s J1 | TestSolrConfigHandlerCloud.test <<<
             [junit4]    > Throwable #1: java.lang.AssertionError: Could not get expected value  'b' for path 'overlay/requestHandler/\/x/a' full output: {
             [junit4]    >   "responseHeader":{
             [junit4]    >     "status":0,
             [junit4]    >     "QTime":0},
             [junit4]    >   "overlay":{
             [junit4]    >     "znodeVersion":0,
             [junit4]    >     "requestHandler":{"/x":{
             [junit4]    >         "name":"/x",
             [junit4]    >         "class":"org.apache.solr.handler.DumpRequestHandler",
             [junit4]    >         "startup":"lazy"}}}}
             [junit4]    > 	at __randomizedtesting.SeedInfo.seed([59310820B646978D:D16537FA18BAFA75]:0)
             [junit4]    > 	at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:399)
             [junit4]    > 	at org.apache.solr.core.TestSolrConfigHandler.reqhandlertests(TestSolrConfigHandler.java:189)
             [junit4]    > 	at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqHandlerAPIs(TestSolrConfigHandlerCloud.java:90)
             [junit4]    > 	at org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:76)
          

          The root cause seems to be the following:

             [junit4]   2> 1014079 T3226 oasc.ZkController$WatcherImpl$1.run WARN listener throws error org.apache.solr.common.SolrException: Unable to reload core [collection1]
             [junit4]   2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:615)
             [junit4]   2> 	at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2696)
             [junit4]   2> 	at org.apache.solr.cloud.ZkController$WatcherImpl$1.run(ZkController.java:2294)
             [junit4]   2> Caused by: org.apache.solr.common.SolrException: Could not load conf for core collection1: Error loading solr config from solrconfig.xml
             [junit4]   2> 	at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:66)
             [junit4]   2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:608)
             [junit4]   2> 	... 2 more
             [junit4]   2> Caused by: org.apache.solr.common.SolrException: Error loading solr config from solrconfig.xml
             [junit4]   2> 	at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:167)
             [junit4]   2> 	at org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:80)
             [junit4]   2> 	at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:61)
             [junit4]   2> 	... 3 more
             [junit4]   2> Caused by: java.io.IOException: Error opening /configs/conf1/solrconfig.xml
             [junit4]   2> 	at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:88)
             [junit4]   2> 	at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:309)
             [junit4]   2> 	at org.apache.solr.core.Config.<init>(Config.java:122)
             [junit4]   2> 	at org.apache.solr.core.Config.<init>(Config.java:92)
             [junit4]   2> 	at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:180)
             [junit4]   2> 	at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:158)
             [junit4]   2> 	... 5 more
             [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /configs/conf1/solrconfig.xml
             [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
             [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
             [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
          

          Core load/reload etc is not well tested in the face of zk session expiration I guess. We should add some (separate) tests for this sort of thing.

          Show
          Shalin Shekhar Mangar added a comment - Another failure from my local jenkins: [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrConfigHandlerCloud -Dtests.method=test -Dtests.seed=59310820B646978D -Dtests.slow= true -Dtests.locale=th_TH -Dtests.timezone=Africa/Sao_Tome -Dtests.asserts= true -Dtests.file.encoding=US-ASCII [junit4] FAILURE 37.5s J1 | TestSolrConfigHandlerCloud.test <<< [junit4] > Throwable #1: java.lang.AssertionError: Could not get expected value 'b' for path 'overlay/requestHandler/\/x/a' full output: { [junit4] > "responseHeader" :{ [junit4] > "status" :0, [junit4] > "QTime" :0}, [junit4] > "overlay" :{ [junit4] > "znodeVersion" :0, [junit4] > "requestHandler" :{ "/x" :{ [junit4] > "name" : "/x" , [junit4] > "class" : "org.apache.solr.handler.DumpRequestHandler" , [junit4] > "startup" : "lazy" }}}} [junit4] > at __randomizedtesting.SeedInfo.seed([59310820B646978D:D16537FA18BAFA75]:0) [junit4] > at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:399) [junit4] > at org.apache.solr.core.TestSolrConfigHandler.reqhandlertests(TestSolrConfigHandler.java:189) [junit4] > at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqHandlerAPIs(TestSolrConfigHandlerCloud.java:90) [junit4] > at org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:76) The root cause seems to be the following: [junit4] 2> 1014079 T3226 oasc.ZkController$WatcherImpl$1.run WARN listener throws error org.apache.solr.common.SolrException: Unable to reload core [collection1] [junit4] 2> at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:615) [junit4] 2> at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2696) [junit4] 2> at org.apache.solr.cloud.ZkController$WatcherImpl$1.run(ZkController.java:2294) [junit4] 2> Caused by: org.apache.solr.common.SolrException: Could not load conf for core collection1: Error loading solr config from solrconfig.xml [junit4] 2> at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:66) [junit4] 2> at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:608) [junit4] 2> ... 2 more [junit4] 2> Caused by: org.apache.solr.common.SolrException: Error loading solr config from solrconfig.xml [junit4] 2> at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:167) [junit4] 2> at org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:80) [junit4] 2> at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:61) [junit4] 2> ... 3 more [junit4] 2> Caused by: java.io.IOException: Error opening /configs/conf1/solrconfig.xml [junit4] 2> at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:88) [junit4] 2> at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:309) [junit4] 2> at org.apache.solr.core.Config.<init>(Config.java:122) [junit4] 2> at org.apache.solr.core.Config.<init>(Config.java:92) [junit4] 2> at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:180) [junit4] 2> at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:158) [junit4] 2> ... 5 more [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /configs/conf1/solrconfig.xml [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) Core load/reload etc is not well tested in the face of zk session expiration I guess. We should add some (separate) tests for this sort of thing.
          Hide
          ASF subversion and git services added a comment -

          Commit 1659860 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1659860 ]

          SOLR-6924: messageformat requires strings

          Show
          ASF subversion and git services added a comment - Commit 1659860 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1659860 ] SOLR-6924 : messageformat requires strings
          Hide
          ASF subversion and git services added a comment -

          Commit 1659861 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1659861 ]

          SOLR-6924: messageformat requires strings

          Show
          ASF subversion and git services added a comment - Commit 1659861 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1659861 ] SOLR-6924 : messageformat requires strings
          Hide
          Yonik Seeley added a comment -

          Just hit one of these failures myself and thought it was my code... ugh.

          Show
          Yonik Seeley added a comment - Just hit one of these failures myself and thought it was my code... ugh.
          Hide
          Noble Paul added a comment -

          Make the watches more robust , so that reconnect will fire even listeners and try reload again

          Show
          Noble Paul added a comment - Make the watches more robust , so that reconnect will fire even listeners and try reload again
          Hide
          ASF subversion and git services added a comment -

          Commit 1669195 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1669195 ]

          SOLR-6924: conf node listening made more robust to take care of session expiry and reconnect

          Show
          ASF subversion and git services added a comment - Commit 1669195 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1669195 ] SOLR-6924 : conf node listening made more robust to take care of session expiry and reconnect
          Hide
          ASF subversion and git services added a comment -

          Commit 1669209 from Noble Paul in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1669209 ]

          SOLR-6924: conf node listening made more robust to take care of session expiry and reconnect

          Show
          ASF subversion and git services added a comment - Commit 1669209 from Noble Paul in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1669209 ] SOLR-6924 : conf node listening made more robust to take care of session expiry and reconnect
          Hide
          ASF subversion and git services added a comment -

          Commit 1669240 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1669240 ]

          SOLR-6924: conf node listening made more robust to take care of session expiry and reconnect

          Show
          ASF subversion and git services added a comment - Commit 1669240 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1669240 ] SOLR-6924 : conf node listening made more robust to take care of session expiry and reconnect
          Hide
          ASF subversion and git services added a comment -

          Commit 1669245 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1669245 ]

          SOLR-6924: removing double listening of nodes

          Show
          ASF subversion and git services added a comment - Commit 1669245 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1669245 ] SOLR-6924 : removing double listening of nodes
          Hide
          ASF subversion and git services added a comment -

          Commit 1669246 from Noble Paul in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1669246 ]

          SOLR-6924: removing the accidental commit. Will do it if a 5.01 comes up

          Show
          ASF subversion and git services added a comment - Commit 1669246 from Noble Paul in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1669246 ] SOLR-6924 : removing the accidental commit. Will do it if a 5.01 comes up
          Hide
          Noble Paul added a comment -

          New fix that would invoke a refresh command to each replica to ensure that they are all updated to the latest config version. This is the same strategy used by schema reloads as well

          Show
          Noble Paul added a comment - New fix that would invoke a refresh command to each replica to ensure that they are all updated to the latest config version. This is the same strategy used by schema reloads as well
          Hide
          Timothy Potter added a comment -

          Seems like waitForAllReplicasState is doing the same thing as ManagedIndexSchema#waitForSchemaZkVersionAgreement? If so, we should fix the code to use 1 implementation vs. having 2 doing the same thing albeit a little differently.

          Show
          Timothy Potter added a comment - Seems like waitForAllReplicasState is doing the same thing as ManagedIndexSchema#waitForSchemaZkVersionAgreement ? If so, we should fix the code to use 1 implementation vs. having 2 doing the same thing albeit a little differently.
          Hide
          ASF subversion and git services added a comment -

          Commit 1670381 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1670381 ]

          SOLR-6924: The config API forcefully refreshes all replicas in the collection to ensure all are
          updated

          Show
          ASF subversion and git services added a comment - Commit 1670381 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1670381 ] SOLR-6924 : The config API forcefully refreshes all replicas in the collection to ensure all are updated
          Hide
          ASF subversion and git services added a comment -

          Commit 1670382 from Noble Paul in branch 'dev/branches/lucene_solr_5_1'
          [ https://svn.apache.org/r1670382 ]

          SOLR-6924: The config API forcefully refreshes all replicas in the collection to ensure all are
          updated

          Show
          ASF subversion and git services added a comment - Commit 1670382 from Noble Paul in branch 'dev/branches/lucene_solr_5_1' [ https://svn.apache.org/r1670382 ] SOLR-6924 : The config API forcefully refreshes all replicas in the collection to ensure all are updated
          Hide
          ASF subversion and git services added a comment -

          Commit 1670383 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1670383 ]

          SOLR-6924: The config API forcefully refreshes all replicas in the collection to ensure all are
          updated

          Show
          ASF subversion and git services added a comment - Commit 1670383 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1670383 ] SOLR-6924 : The config API forcefully refreshes all replicas in the collection to ensure all are updated
          Hide
          Timothy Potter added a comment -

          Bulk close after 5.1 release

          Show
          Timothy Potter added a comment - Bulk close after 5.1 release

            People

            • Assignee:
              Noble Paul
              Reporter:
              Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development