Solr
  1. Solr
  2. SOLR-4982

Creating a core while referencing system properties looks like it loses files.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.3, 5.0
    • Fix Version/s: 4.4, 5.0
    • Component/s: multicore
    • Labels:
      None

      Description

      If you use the core admin handler to create core and reference system properties and index files without restarting Solr, your files are indexed to the wrong place.

      Say for instance I define a sys prop EOE=/Users/Erick/tmp and create a core with this request
      localhost:8983/solr/admin/cores?action=CREATE&name=coreZ&instanceDir=coreZ&dataDir=%24%7BEOE%7D

      where %24%7BEOE%7D is really $

      {EOE} after URL escaping. What gets preserved in solr.xml is correct, dataDir is set to ${EOE}

      . And if I restart Solr, then index documents, they wind up in /Users/Erick/tmp. This is as it should be.

      HOWEVER, if rather than immediately restart Solr I index some documents to CoreZ, they go in <solr_home>/CoreZ/$

      {EOE}. The literal path is ${EOE}

      , dollar sign, curly braces and all.

      How important is this to fix for 4.4?

      1. SOLR-4982.patch
        8 kB
        Erick Erickson
      2. SOLR-4982.patch
        20 kB
        Erick Erickson
      3. SOLR-4982.patch
        23 kB
        Erick Erickson
      4. SOLR-4982.patch
        23 kB
        Erick Erickson
      5. SOLR-4982-fixtest.patch
        1 kB
        Erick Erickson

        Issue Links

          Activity

          Hide
          Erick Erickson added a comment -

          Patch with test case and fixes. All other tests pass.

          I'll check this in later today, I want to do a few more tests. Unless there are objections.

          Show
          Erick Erickson added a comment - Patch with test case and fixes. All other tests pass. I'll check this in later today, I want to do a few more tests. Unless there are objections.
          Hide
          Erick Erickson added a comment -

          Along the way, I wondered "gee, what happens if we create a core in discovery mode?" Well, we don't preserve properties passed on the URL. This patch preserves any parameters passed in on the admin URL, e.g. dataDir, config etc.

          Oddly, you have to specify instanceDir even though it isn't a valid property for core.properties, else how do we let the user specify something not immediately below <solrhome>?

          But my remaining problem is that I can't exit the test gracefully, searchers are left hanging and we get the partial stack trace below.

          This usually means we did a CoreContainer.get and no corresponding close, but there aren't any such things. I think I even closed loading properties this time, although that would be unrelated I think.

          When we create cores in old-style solr.xml land, there's no need to do anything special to exit the tests. So I'm hoping someone has an "aha" moment. Otherwise I need to dig into core creation in the discovery case and find out if it really is different.

          java.lang.AssertionError: ERROR: SolrIndexSearcher opens=3 closes=1
          at __randomizedtesting.SeedInfo.seed([445018872D608FD5]:0)
          at org.junit.Assert.fail(Assert.java:93)
          at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:275)
          at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:122)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
          at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)

          Show
          Erick Erickson added a comment - Along the way, I wondered "gee, what happens if we create a core in discovery mode?" Well, we don't preserve properties passed on the URL. This patch preserves any parameters passed in on the admin URL, e.g. dataDir, config etc. Oddly, you have to specify instanceDir even though it isn't a valid property for core.properties, else how do we let the user specify something not immediately below <solrhome>? But my remaining problem is that I can't exit the test gracefully, searchers are left hanging and we get the partial stack trace below. This usually means we did a CoreContainer.get and no corresponding close, but there aren't any such things. I think I even closed loading properties this time, although that would be unrelated I think. When we create cores in old-style solr.xml land, there's no need to do anything special to exit the tests. So I'm hoping someone has an "aha" moment. Otherwise I need to dig into core creation in the discovery case and find out if it really is different. java.lang.AssertionError: ERROR: SolrIndexSearcher opens=3 closes=1 at __randomizedtesting.SeedInfo.seed( [445018872D608FD5] :0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:275) at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:122) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
          Hide
          Erick Erickson added a comment -

          Latest version. Thanks to Shalin I managed to find a way to have the test terminate.

          Along the way I've added a new method to SolrTestCaseJ4 that creates a "no core" test harness that assumes discovery mode on the theory that it took me a while to figure out how to do that and we were going to do this a lot more in the future. Any easier ways to do this?

          Running tests and doing some manual inspection. If that all works out I'll probably check this in today.

          Show
          Erick Erickson added a comment - Latest version. Thanks to Shalin I managed to find a way to have the test terminate. Along the way I've added a new method to SolrTestCaseJ4 that creates a "no core" test harness that assumes discovery mode on the theory that it took me a while to figure out how to do that and we were going to do this a lot more in the future. Any easier ways to do this? Running tests and doing some manual inspection. If that all works out I'll probably check this in today.
          Hide
          Erick Erickson added a comment -

          Alan Woodward Your changes to SOLR-4948 turned out to be something that I used (unknowingly) for this patch. When do you expect to merge 4948 with 4x? I'd like to get this in to 4.4, but don't want to rush things unduly.

          Thanks,
          Erick

          Show
          Erick Erickson added a comment - Alan Woodward Your changes to SOLR-4948 turned out to be something that I used (unknowingly) for this patch. When do you expect to merge 4948 with 4x? I'd like to get this in to 4.4, but don't want to rush things unduly. Thanks, Erick
          Hide
          Erick Erickson added a comment -

          Turning to a blocker so I don't lose track of it for 4.4. Need o get SOLR-4948 checked in to untangle the test harness so I can invoke it cleanly with no cores defined in discovery mode. Maybe there's a better way?

          Show
          Erick Erickson added a comment - Turning to a blocker so I don't lose track of it for 4.4. Need o get SOLR-4948 checked in to untangle the test harness so I can invoke it cleanly with no cores defined in discovery mode. Maybe there's a better way?
          Hide
          Erick Erickson added a comment -

          Final patch, committing momentarily

          Show
          Erick Erickson added a comment - Final patch, committing momentarily
          Hide
          ASF subversion and git services added a comment -

          Commit 1500284 from Erick Erickson
          [ https://svn.apache.org/r1500284 ]

          Fix for SOLR-4982, creating cores with sysprops does not dereference them properly

          Show
          ASF subversion and git services added a comment - Commit 1500284 from Erick Erickson [ https://svn.apache.org/r1500284 ] Fix for SOLR-4982 , creating cores with sysprops does not dereference them properly
          Hide
          Erick Erickson added a comment -

          trunk: r - 1500284

          Needs to wait on a couple of other patches to be merged into 4.x to commit there.

          Show
          Erick Erickson added a comment - trunk: r - 1500284 Needs to wait on a couple of other patches to be merged into 4.x to commit there.
          Hide
          Erick Erickson added a comment -

          An attempt to fix the broken test.

          Apply after main patch.

          Show
          Erick Erickson added a comment - An attempt to fix the broken test. Apply after main patch.
          Hide
          ASF subversion and git services added a comment -

          Commit 1500354 from Erick Erickson
          [ https://svn.apache.org/r1500354 ]

          Fix broken test checked in with SOLR-4982

          Show
          ASF subversion and git services added a comment - Commit 1500354 from Erick Erickson [ https://svn.apache.org/r1500354 ] Fix broken test checked in with SOLR-4982
          Hide
          ASF subversion and git services added a comment -

          Commit 1500359 from Erick Erickson
          [ https://svn.apache.org/r1500359 ]

          SOLR-4982 4x checkin, including the test fix (trunk r 1500354)

          Show
          ASF subversion and git services added a comment - Commit 1500359 from Erick Erickson [ https://svn.apache.org/r1500359 ] SOLR-4982 4x checkin, including the test fix (trunk r 1500354)
          Hide
          Steve Rowe added a comment -

          Bulk close resolved 4.4 issues

          Show
          Steve Rowe added a comment - Bulk close resolved 4.4 issues

            People

            • Assignee:
              Erick Erickson
              Reporter:
              Erick Erickson
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development