Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-4982

Creating a core while referencing system properties looks like it loses files.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.3, 6.0
    • Fix Version/s: 4.4, 6.0
    • Component/s: multicore
    • Labels:
      None

      Description

      If you use the core admin handler to create core and reference system properties and index files without restarting Solr, your files are indexed to the wrong place.

      Say for instance I define a sys prop EOE=/Users/Erick/tmp and create a core with this request
      localhost:8983/solr/admin/cores?action=CREATE&name=coreZ&instanceDir=coreZ&dataDir=%24%7BEOE%7D

      where %24%7BEOE%7D is really $

      {EOE} after URL escaping. What gets preserved in solr.xml is correct, dataDir is set to ${EOE}

      . And if I restart Solr, then index documents, they wind up in /Users/Erick/tmp. This is as it should be.

      HOWEVER, if rather than immediately restart Solr I index some documents to CoreZ, they go in <solr_home>/CoreZ/$

      {EOE}. The literal path is ${EOE}

      , dollar sign, curly braces and all.

      How important is this to fix for 4.4?

      1. SOLR-4982.patch
        8 kB
        Erick Erickson
      2. SOLR-4982.patch
        20 kB
        Erick Erickson
      3. SOLR-4982.patch
        23 kB
        Erick Erickson
      4. SOLR-4982.patch
        23 kB
        Erick Erickson
      5. SOLR-4982-fixtest.patch
        1 kB
        Erick Erickson

        Issue Links

          Activity

          Hide
          steve_rowe Steve Rowe added a comment -

          Bulk close resolved 4.4 issues

          Show
          steve_rowe Steve Rowe added a comment - Bulk close resolved 4.4 issues
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1500359 from Erick Erickson
          [ https://svn.apache.org/r1500359 ]

          SOLR-4982 4x checkin, including the test fix (trunk r 1500354)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1500359 from Erick Erickson [ https://svn.apache.org/r1500359 ] SOLR-4982 4x checkin, including the test fix (trunk r 1500354)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1500354 from Erick Erickson
          [ https://svn.apache.org/r1500354 ]

          Fix broken test checked in with SOLR-4982

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1500354 from Erick Erickson [ https://svn.apache.org/r1500354 ] Fix broken test checked in with SOLR-4982
          Hide
          erickerickson Erick Erickson added a comment -

          An attempt to fix the broken test.

          Apply after main patch.

          Show
          erickerickson Erick Erickson added a comment - An attempt to fix the broken test. Apply after main patch.
          Hide
          erickerickson Erick Erickson added a comment -

          trunk: r - 1500284

          Needs to wait on a couple of other patches to be merged into 4.x to commit there.

          Show
          erickerickson Erick Erickson added a comment - trunk: r - 1500284 Needs to wait on a couple of other patches to be merged into 4.x to commit there.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1500284 from Erick Erickson
          [ https://svn.apache.org/r1500284 ]

          Fix for SOLR-4982, creating cores with sysprops does not dereference them properly

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1500284 from Erick Erickson [ https://svn.apache.org/r1500284 ] Fix for SOLR-4982 , creating cores with sysprops does not dereference them properly
          Hide
          erickerickson Erick Erickson added a comment -

          Final patch, committing momentarily

          Show
          erickerickson Erick Erickson added a comment - Final patch, committing momentarily
          Hide
          erickerickson Erick Erickson added a comment -

          Turning to a blocker so I don't lose track of it for 4.4. Need o get SOLR-4948 checked in to untangle the test harness so I can invoke it cleanly with no cores defined in discovery mode. Maybe there's a better way?

          Show
          erickerickson Erick Erickson added a comment - Turning to a blocker so I don't lose track of it for 4.4. Need o get SOLR-4948 checked in to untangle the test harness so I can invoke it cleanly with no cores defined in discovery mode. Maybe there's a better way?
          Hide
          erickerickson Erick Erickson added a comment -

          Alan Woodward Your changes to SOLR-4948 turned out to be something that I used (unknowingly) for this patch. When do you expect to merge 4948 with 4x? I'd like to get this in to 4.4, but don't want to rush things unduly.

          Thanks,
          Erick

          Show
          erickerickson Erick Erickson added a comment - Alan Woodward Your changes to SOLR-4948 turned out to be something that I used (unknowingly) for this patch. When do you expect to merge 4948 with 4x? I'd like to get this in to 4.4, but don't want to rush things unduly. Thanks, Erick
          Hide
          erickerickson Erick Erickson added a comment -

          Latest version. Thanks to Shalin I managed to find a way to have the test terminate.

          Along the way I've added a new method to SolrTestCaseJ4 that creates a "no core" test harness that assumes discovery mode on the theory that it took me a while to figure out how to do that and we were going to do this a lot more in the future. Any easier ways to do this?

          Running tests and doing some manual inspection. If that all works out I'll probably check this in today.

          Show
          erickerickson Erick Erickson added a comment - Latest version. Thanks to Shalin I managed to find a way to have the test terminate. Along the way I've added a new method to SolrTestCaseJ4 that creates a "no core" test harness that assumes discovery mode on the theory that it took me a while to figure out how to do that and we were going to do this a lot more in the future. Any easier ways to do this? Running tests and doing some manual inspection. If that all works out I'll probably check this in today.
          Hide
          erickerickson Erick Erickson added a comment -

          Along the way, I wondered "gee, what happens if we create a core in discovery mode?" Well, we don't preserve properties passed on the URL. This patch preserves any parameters passed in on the admin URL, e.g. dataDir, config etc.

          Oddly, you have to specify instanceDir even though it isn't a valid property for core.properties, else how do we let the user specify something not immediately below <solrhome>?

          But my remaining problem is that I can't exit the test gracefully, searchers are left hanging and we get the partial stack trace below.

          This usually means we did a CoreContainer.get and no corresponding close, but there aren't any such things. I think I even closed loading properties this time, although that would be unrelated I think.

          When we create cores in old-style solr.xml land, there's no need to do anything special to exit the tests. So I'm hoping someone has an "aha" moment. Otherwise I need to dig into core creation in the discovery case and find out if it really is different.

          java.lang.AssertionError: ERROR: SolrIndexSearcher opens=3 closes=1
          at __randomizedtesting.SeedInfo.seed([445018872D608FD5]:0)
          at org.junit.Assert.fail(Assert.java:93)
          at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:275)
          at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:122)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
          at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)

          Show
          erickerickson Erick Erickson added a comment - Along the way, I wondered "gee, what happens if we create a core in discovery mode?" Well, we don't preserve properties passed on the URL. This patch preserves any parameters passed in on the admin URL, e.g. dataDir, config etc. Oddly, you have to specify instanceDir even though it isn't a valid property for core.properties, else how do we let the user specify something not immediately below <solrhome>? But my remaining problem is that I can't exit the test gracefully, searchers are left hanging and we get the partial stack trace below. This usually means we did a CoreContainer.get and no corresponding close, but there aren't any such things. I think I even closed loading properties this time, although that would be unrelated I think. When we create cores in old-style solr.xml land, there's no need to do anything special to exit the tests. So I'm hoping someone has an "aha" moment. Otherwise I need to dig into core creation in the discovery case and find out if it really is different. java.lang.AssertionError: ERROR: SolrIndexSearcher opens=3 closes=1 at __randomizedtesting.SeedInfo.seed( [445018872D608FD5] :0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:275) at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:122) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
          Hide
          erickerickson Erick Erickson added a comment -

          Patch with test case and fixes. All other tests pass.

          I'll check this in later today, I want to do a few more tests. Unless there are objections.

          Show
          erickerickson Erick Erickson added a comment - Patch with test case and fixes. All other tests pass. I'll check this in later today, I want to do a few more tests. Unless there are objections.

            People

            • Assignee:
              erickerickson Erick Erickson
              Reporter:
              erickerickson Erick Erickson
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development