Solr
  1. Solr
  2. SOLR-7470

jvm/filesystem dependent failures in SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection -- directory traversal order dependency

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.2, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection currently fails 100% reliably on my laptop, regardless of seed with a root cause of...

         [junit4]   2> 35968 T60 N:127.0.0.1:57372_ C:gettingstarted S:shard1 R:core_node1 c:gettingstarted_shard1_replica2 C15 oasc.SolrException.log ERROR org.apache.solr.common.SolrException: ERROR: [doc=VS1GB400C3] Error adding field 'price'='74.99' msg=For input string: "74.99"
         [junit4]   2> 		at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:178)
         [junit4]   2> 		at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:79)
         [junit4]   2> 		at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:237)
         [junit4]   2> 		at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
         [junit4]   2> 		at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
      

      This test uses "data_driven_schema_configs" and indexes all of the *.xml files in "example/exampledocs".

      Two factors seem to be contributing to the the reason this fails consistently for me (but not for jenkins or anyone else i've seen complain)...

      1. The File.listFiles(FileFilter) method is used to iterate over the files
      2. The "mem.xml" file has an integer price value: <field name="price">185</field>

      listFiles is documented that "There is no guarantee that the name strings in the resulting array will appear in any specific order" and evidently with my filesystem + JVM they come back in a consistent order everytime, which just so happens to put mem.xml in front of any other file that also has a "price" field.

        Issue Links

          Activity

          Hide
          Hoss Man added a comment - - edited

          Attached patch:

          • fixes the test to index the xml files in a deterministic order dictated by the test seed.
          • fixes mem.xml to use float price value.

          If you apply the patch, and then revert the mem.xml change, anyone – regardless of filesystem – should be able to see the test reliably fail with this seed...

          ant test  -Dtestcase=SolrCloudExampleTest -Dtests.method=testLoadDocsIntoGettingStartedCollection -Dtests.seed=2AD197B874223638:C770325C8CC3DD07 -Dtests.slow=true -Dtests.locale=es_PE -Dtests.timezone=Africa/Khartoum -Dtests.asserts=true -Dtests.file.encoding=UTF-8
          

          ...i'm still hammering on the test to verify there are no attempt to identify other file orderings that trigger similar bugs.

          Show
          Hoss Man added a comment - - edited Attached patch: fixes the test to index the xml files in a deterministic order dictated by the test seed. fixes mem.xml to use float price value. If you apply the patch, and then revert the mem.xml change, anyone – regardless of filesystem – should be able to see the test reliably fail with this seed... ant test -Dtestcase=SolrCloudExampleTest -Dtests.method=testLoadDocsIntoGettingStartedCollection -Dtests.seed=2AD197B874223638:C770325C8CC3DD07 -Dtests.slow=true -Dtests.locale=es_PE -Dtests.timezone=Africa/Khartoum -Dtests.asserts=true -Dtests.file.encoding=UTF-8 ...i'm still hammering on the test to verify there are no attempt to identify other file orderings that trigger similar bugs.
          Hide
          ASF subversion and git services added a comment -

          Commit 1676337 from hossman@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1676337 ]

          SOLR-7470: Fix sample data to eliminate file order dependency for successful indexing, also fixed SolrCloudExampleTest to help catch this in the future

          Show
          ASF subversion and git services added a comment - Commit 1676337 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1676337 ] SOLR-7470 : Fix sample data to eliminate file order dependency for successful indexing, also fixed SolrCloudExampleTest to help catch this in the future
          Hide
          ASF subversion and git services added a comment -

          Commit 1676342 from hossman@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1676342 ]

          SOLR-7470: Fix sample data to eliminate file order dependency for successful indexing, also fixed SolrCloudExampleTest to help catch this in the future (merge r1676337)

          Show
          ASF subversion and git services added a comment - Commit 1676342 from hossman@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1676342 ] SOLR-7470 : Fix sample data to eliminate file order dependency for successful indexing, also fixed SolrCloudExampleTest to help catch this in the future (merge r1676337)
          Hide
          Hoss Man added a comment -

          whoops ... aparently i forgot to close this the other day

          (committed fix after running a 500 random iterations w/o seeing any similar types of failures)

          Show
          Hoss Man added a comment - whoops ... aparently i forgot to close this the other day (committed fix after running a 500 random iterations w/o seeing any similar types of failures)
          Hide
          Anshum Gupta added a comment -

          Bulk close for 5.2.0.

          Show
          Anshum Gupta added a comment - Bulk close for 5.2.0.

            People

            • Assignee:
              Hoss Man
              Reporter:
              Hoss Man
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development