Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10719

Creating a core.properties fails if the parent of core.properties is a symlinked dierctory

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.6.1, 7.0, master (8.0)
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Well, it doesn't actually fail until you try to restart the Solr instance. The root is that creating core.properties fails.

      This is due to SOLR-8260. CorePropertiesLocator.writePropertiesFile changed from:
      propfile.getParentFile().mkdirs();
      to
      Files.createDirectories(propfile.getParent());

      The former (apparently) thinks it's OK if a symlink points to a directory, but the latter throws an exception.

      So the behavior here is that the call appears to succeed, the replica is created and is functional. Until you restart the instance when it's not discovered.

      I hacked in a simple test to see if the parent existed already and skip the call to createDirectories if so and ADDREPLICA works just fine. Restarting Solr finds the replica.

      The test "for real" would probably have to be better than this as we probably really want to keep from overwriting an existing replica and the like, didn't check whether that's already accounted for though.

      There's another issue here that failing to write the properties file should fail the ADDREPLICA IMO.

      Alan Woodward I'm guessing that this is an unintended side-effect of SOLR-8260 but wanted to check before diving in deeper.

      1. SOLR-10719.patch
        3 kB
        Erick Erickson
      2. SOLR-10719.patch
        3 kB
        Erick Erickson

        Activity

        Hide
        erickerickson Erick Erickson added a comment -

        Assigning to myself to track, but anyone else should feel free to tackle it.

        Show
        erickerickson Erick Erickson added a comment - Assigning to myself to track, but anyone else should feel free to tackle it.
        Hide
        romseygeek Alan Woodward added a comment -

        I don't follow what's happening here - are you saying that Files.createDirectories() fails, but the ADDREPLICA call succeeds anyway? That sounds ... wrong?

        Show
        romseygeek Alan Woodward added a comment - I don't follow what's happening here - are you saying that Files.createDirectories() fails, but the ADDREPLICA call succeeds anyway? That sounds ... wrong?
        Hide
        erickerickson Erick Erickson added a comment -

        Alan Woodward There's no question that this is wrong. Yes, the ADDREPLICA succeeds. I looked at it quickly and the replica is even up and running. Until you restart solr then core.properties is not found, the core is not discovered and people are left wondering what happened. If you manually reconstruct the core.properties file at the destination of the symlink and restart, the core is alive and healthy. This is not theoretical BTW, it happened "in the field".

        It's particularly scary when you then DELETEREPLICA because you see your new replica active. It takes over leadership and everything.

        This worked in earlier versions of Solr.

        Files.createDirectories throws an exception "file already exists" IIRC, but it's swallowed, doesn't propagate back up the stack.

        So there are two issues here I suppose:
        1> should we follow symlinks (I think we should)
        2> should failing to write core.properties files cause the ADDREPLICA to fail (I think it should).

        Maybe break <2> out into a new JIRA?

        Show
        erickerickson Erick Erickson added a comment - Alan Woodward There's no question that this is wrong. Yes, the ADDREPLICA succeeds. I looked at it quickly and the replica is even up and running. Until you restart solr then core.properties is not found, the core is not discovered and people are left wondering what happened. If you manually reconstruct the core.properties file at the destination of the symlink and restart, the core is alive and healthy. This is not theoretical BTW, it happened "in the field". It's particularly scary when you then DELETEREPLICA because you see your new replica active. It takes over leadership and everything. This worked in earlier versions of Solr. Files.createDirectories throws an exception "file already exists" IIRC, but it's swallowed, doesn't propagate back up the stack. So there are two issues here I suppose: 1> should we follow symlinks (I think we should) 2> should failing to write core.properties files cause the ADDREPLICA to fail (I think it should). Maybe break <2> out into a new JIRA?
        Hide
        erickerickson Erick Erickson added a comment - - edited

        This is actually a bit weirder. Files.createDirectories does succeed if you specify a subdir of the symlink. So if
        sym -> dir1

        and I ask CreateDirectories to create sym/eoe1/eoe2/eoe3 all the directories are created just fine. But when core.properties is being written, it wants to write to sym/core.properties and the createDirectories fails on creating sym as it's a symlink.

        I see multiple places in the code where we call Files.createDirectories, even some tagged with

        //note, this will fail if this is a symlink

        All in all, symlinks are going to be a problem in several places in the code.

        So I'm thinking of providing a method in FileUtils to deal with this kind of thing that would then be available for other users as appropriate.

        Oh, and I'm not suggesting that we make this a blanket change as I'm not sure these other places should be changed.

        Show
        erickerickson Erick Erickson added a comment - - edited This is actually a bit weirder. Files.createDirectories does succeed if you specify a subdir of the symlink. So if sym -> dir1 and I ask CreateDirectories to create sym/eoe1/eoe2/eoe3 all the directories are created just fine. But when core.properties is being written, it wants to write to sym/core.properties and the createDirectories fails on creating sym as it's a symlink. I see multiple places in the code where we call Files.createDirectories, even some tagged with //note, this will fail if this is a symlink All in all, symlinks are going to be a problem in several places in the code. So I'm thinking of providing a method in FileUtils to deal with this kind of thing that would then be available for other users as appropriate. Oh, and I'm not suggesting that we make this a blanket change as I'm not sure these other places should be changed.
        Hide
        erickerickson Erick Erickson added a comment -

        First cut at a patch. It seems to handle the cases in this JIRA, i.e. if the dest is a symlink it'll still create the core and write the core.properties. Additionally, if the core.properties file cannot be created it throws an error.

        Will look more tomorrow but so far this approach looks promising.

        Show
        erickerickson Erick Erickson added a comment - First cut at a patch. It seems to handle the cases in this JIRA, i.e. if the dest is a symlink it'll still create the core and write the core.properties. Additionally, if the core.properties file cannot be created it throws an error. Will look more tomorrow but so far this approach looks promising.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 412e4ae2c192a5b444dd63220e5918f4e7fd47be in lucene-solr's branch refs/heads/master from Erick Erickson
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=412e4ae ]

        SOLR-10719: Creating a core.properties fails if the parent of core.properties is a symlinked dierctory

        Show
        jira-bot ASF subversion and git services added a comment - Commit 412e4ae2c192a5b444dd63220e5918f4e7fd47be in lucene-solr's branch refs/heads/master from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=412e4ae ] SOLR-10719 : Creating a core.properties fails if the parent of core.properties is a symlinked dierctory
        Hide
        erickerickson Erick Erickson added a comment -

        final patch with CHANGES

        Show
        erickerickson Erick Erickson added a comment - final patch with CHANGES
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ee10c452d1710ad2b0028675623dfee467981afd in lucene-solr's branch refs/heads/branch_6x from Erick Erickson
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ee10c45 ]

        SOLR-10719: Creating a core.properties fails if the parent of core.properties is a symlinked dierctory

        (cherry picked from commit 412e4ae)

        Show
        jira-bot ASF subversion and git services added a comment - Commit ee10c452d1710ad2b0028675623dfee467981afd in lucene-solr's branch refs/heads/branch_6x from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ee10c45 ] SOLR-10719 : Creating a core.properties fails if the parent of core.properties is a symlinked dierctory (cherry picked from commit 412e4ae)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 425af4f658de763821fea41b763fb3fda8316ad0 in lucene-solr's branch refs/heads/branch_6_6 from Erick Erickson
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=425af4f ]

        SOLR-10719: Creating a core.properties fails if the parent of core.properties is a symlinked dierctory

        (cherry picked from commit ee10c45)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 425af4f658de763821fea41b763fb3fda8316ad0 in lucene-solr's branch refs/heads/branch_6_6 from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=425af4f ] SOLR-10719 : Creating a core.properties fails if the parent of core.properties is a symlinked dierctory (cherry picked from commit ee10c45)

          People

          • Assignee:
            erickerickson Erick Erickson
            Reporter:
            erickerickson Erick Erickson
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development