Solr
  1. Solr
  2. SOLR-5323

Solr requires -Dsolr.clustering.enabled=false when pointing at example config

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.5
    • Fix Version/s: 4.5.1, 4.6, 6.0
    • Component/s: contrib - Clustering
    • Labels:
      None
    • Environment:

      vanilla mac

      Description

      my typical use of Solr is something like this:

      cd SOLR_HOME/example
      cp -r solr /myProjectDir/solr_home
      java -jar -Dsolr.solr.home=/myProjectDir/solr_home  start.jar
      

      But in solr 4.5.0 this fails to start successfully. I get an error:

      org.apache.solr.common.SolrException: Error loading class 'solr.clustering.ClusteringComponent'
      

      The reason is because solr.clustering.enabled defaults to true now. I don't know why this might be the case.

      you can get around it with

      java -jar -Dsolr.solr.home=/myProjectDir/solr_home -Dsolr.clustering.enabled=false start.jar
      

      SOLR-4708 is when this became an issue.

        Activity

        Hide
        Erik Hatcher added a comment -

        I think we should have the <lib> elements in solrconfig.xml be something like this:

          <lib dir="${solr.install.dir}/contrib/clustering/lib/" regex=".*\.jar" />
        

        where solr.install.dir is a property defined by Solr automatically at startup that has the root of where Solr is installed. I've done this manually by adjusting the configuration in this exact scenario (copying the example configuration and changing all <lib>'s in this way and defining solr.install.dir on the command-line), but Solr should be able to do this better.

        Show
        Erik Hatcher added a comment - I think we should have the <lib> elements in solrconfig.xml be something like this: <lib dir= "${solr.install.dir}/contrib/clustering/lib/" regex= ".*\.jar" /> where solr.install.dir is a property defined by Solr automatically at startup that has the root of where Solr is installed. I've done this manually by adjusting the configuration in this exact scenario (copying the example configuration and changing all <lib>'s in this way and defining solr.install.dir on the command-line), but Solr should be able to do this better.
        Hide
        Erik Hatcher added a comment -

        This isn't specific to the clustering component, except that it gets loaded non-lazily. See these comments: https://issues.apache.org/jira/browse/SOLR-4708?focusedCommentId=13630567&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13630567

        Show
        Erik Hatcher added a comment - This isn't specific to the clustering component, except that it gets loaded non-lazily. See these comments: https://issues.apache.org/jira/browse/SOLR-4708?focusedCommentId=13630567&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13630567
        Hide
        Yonik Seeley added a comment -

        Hmmm, I agree this is a bug.
        My comment in SOLR-4708 was "+1, provided that everything (except clustering) still works if you copy "example" somewhere else."

        Show
        Yonik Seeley added a comment - Hmmm, I agree this is a bug. My comment in SOLR-4708 was "+1, provided that everything (except clustering) still works if you copy "example" somewhere else."
        Hide
        Erik Hatcher added a comment -

        My comment in SOLR-4708 was "+1, provided that everything (except clustering) still works if you copy "example" somewhere else."

        And that's the reason I didn't commit it before. I thought somehow Dawid had worked some magic to alleviate this issue when he took it on.

        We should perhaps have lazy loaded SearchComponents too?

        Show
        Erik Hatcher added a comment - My comment in SOLR-4708 was "+1, provided that everything (except clustering) still works if you copy "example" somewhere else." And that's the reason I didn't commit it before. I thought somehow Dawid had worked some magic to alleviate this issue when he took it on. We should perhaps have lazy loaded SearchComponents too?
        Hide
        Dawid Weiss added a comment -

        I can't remember but I think the problem was that it wasn't possible to define install-dir relative directories for lib element. I'll take a look.

        Show
        Dawid Weiss added a comment - I can't remember but I think the problem was that it wasn't possible to define install-dir relative directories for lib element. I'll take a look.
        Hide
        Dawid Weiss added a comment -

        Hmmm, I agree this is a bug.

        Ok, I admit I don't know what is the "right" way to fix this. I also don't think this issue applies to the clustering contrib only.

        There are several <lib/> elements in the example configuration and they all use relative paths to load additional components. So after you change the relative folder structure any "example" using these will fail to load, the clustering component just happens to be loaded eagerly so it triggers a failure sooner.

        I've had real life use cases when hard-coded relative paths proved very confusing so I'd opt for something like Erik mentioned – making <lib/> components reference JARs relative to some notion of an "installation" folder. Ideally, this should also be inferred automatically if it's not given on command line and if somebody runs Solr from the unpacked default structure. I can prepare a patch that will try to guess and set "solr.install.dir" based on the classloader URI leading to one of the core classes (SolrCore, for example). Once we locate the core JAR we can sniff for ../contrib and ../dist and if these are present set the install dir in the upper folder.

        This is, obviously, a hack. If somebody runs from a WAR file or changes the default directory structure things will break. I assume if they do so they'll also be able to change the example's default paths (or override solr.install.dir explicitly).

        I'm also not 100% confident how the above will work in the distributed setting – soliciting feedback here.

        Anyway, if the above sound all right I will prepare a patch. I don't have any sensible alternatives other than turning off the clustering component by default. I would be against this, however, because like I said – this only hides the problem, not solves it (the default configuration will still be "unmovable" from the default directory structure).

        Show
        Dawid Weiss added a comment - Hmmm, I agree this is a bug. Ok, I admit I don't know what is the "right" way to fix this. I also don't think this issue applies to the clustering contrib only. There are several <lib/> elements in the example configuration and they all use relative paths to load additional components. So after you change the relative folder structure any "example" using these will fail to load, the clustering component just happens to be loaded eagerly so it triggers a failure sooner. I've had real life use cases when hard-coded relative paths proved very confusing so I'd opt for something like Erik mentioned – making <lib/> components reference JARs relative to some notion of an "installation" folder. Ideally, this should also be inferred automatically if it's not given on command line and if somebody runs Solr from the unpacked default structure. I can prepare a patch that will try to guess and set "solr.install.dir" based on the classloader URI leading to one of the core classes (SolrCore, for example). Once we locate the core JAR we can sniff for ../contrib and ../dist and if these are present set the install dir in the upper folder. This is, obviously, a hack. If somebody runs from a WAR file or changes the default directory structure things will break. I assume if they do so they'll also be able to change the example's default paths (or override solr.install.dir explicitly). I'm also not 100% confident how the above will work in the distributed setting – soliciting feedback here. Anyway, if the above sound all right I will prepare a patch. I don't have any sensible alternatives other than turning off the clustering component by default. I would be against this, however, because like I said – this only hides the problem, not solves it (the default configuration will still be "unmovable" from the default directory structure).
        Hide
        Mark Miller added a comment -

        I also think this was a mistake - I don't know that we need another solr.home type thing to address it though.

        The root of the issue is that the clustering is not "really" lazy loading clustering - and the current policy is to lazy load the contrib modules - and that is because of the component. I think Erik is on to the right path with lazy SearchComponents. I think that if the only request handlers that refer to a search component are lazy, they should probably also init lazily. I have not looked into how hard that is to do, but it seems like the correct fix to bring clustering in line with the other contribs. I also think the whole enabled flag we had is no good.

        Show
        Mark Miller added a comment - I also think this was a mistake - I don't know that we need another solr.home type thing to address it though. The root of the issue is that the clustering is not "really" lazy loading clustering - and the current policy is to lazy load the contrib modules - and that is because of the component. I think Erik is on to the right path with lazy SearchComponents. I think that if the only request handlers that refer to a search component are lazy, they should probably also init lazily. I have not looked into how hard that is to do, but it seems like the correct fix to bring clustering in line with the other contribs. I also think the whole enabled flag we had is no good.
        Hide
        Dawid Weiss added a comment -

        I can revert to lazy-loading, not a problem. But this isn't solving the relative paths issue at all. Like I mentioned there were several times when I had to pass an example preconfigured solr configuration to somebody – this always required that person to put the content of the example under a specific directory in Solr distribution, otherwise things wouldn't work because of relative paths. It was a pain to explain why this step is needed and to enforce... I ended up just copying the required JARs into the example. This seems wrong somehow – if it's a solr distribution then there should be a way to reference contribs in a way that allows people to have their stuff in any folder hierarchy?

        What do you think?

        Show
        Dawid Weiss added a comment - I can revert to lazy-loading, not a problem. But this isn't solving the relative paths issue at all. Like I mentioned there were several times when I had to pass an example preconfigured solr configuration to somebody – this always required that person to put the content of the example under a specific directory in Solr distribution, otherwise things wouldn't work because of relative paths. It was a pain to explain why this step is needed and to enforce... I ended up just copying the required JARs into the example. This seems wrong somehow – if it's a solr distribution then there should be a way to reference contribs in a way that allows people to have their stuff in any folder hierarchy? What do you think?
        Hide
        Mark Miller added a comment -

        I just think anything with the relative paths is a separate issue.

        You can use any hierarchy - you just have to change those paths. I'm all for that being improved somehow, but the issue here seems to be:

        Solr contrib modules are lazy loaded so that if you don't use them, you can delete any of them from the dist package layout and things still work. Or you can not delete them and if you try and use them, things work. Clustering now violates that. It's not really clusterings fault, it seems to more be a limitation of the search component.

        Show
        Mark Miller added a comment - I just think anything with the relative paths is a separate issue. You can use any hierarchy - you just have to change those paths. I'm all for that being improved somehow, but the issue here seems to be: Solr contrib modules are lazy loaded so that if you don't use them, you can delete any of them from the dist package layout and things still work. Or you can not delete them and if you try and use them, things work. Clustering now violates that. It's not really clusterings fault, it seems to more be a limitation of the search component.
        Hide
        Dawid Weiss added a comment -

        Ok, I will reverting the changes from SOLR-4708.

        Show
        Dawid Weiss added a comment - Ok, I will reverting the changes from SOLR-4708 .
        Hide
        Dawid Weiss added a comment -

        Patch reverting (portions) of SOLR-4708.

        Show
        Dawid Weiss added a comment - Patch reverting (portions) of SOLR-4708 .
        Hide
        ASF subversion and git services added a comment -

        Commit 1531377 from Dawid Weiss in branch 'dev/trunk'
        [ https://svn.apache.org/r1531377 ]

        SOLR-5323: Disable ClusteringComponent by default in collection1 example. The solr.clustering.enabled system property needs to be set to 'true' to enable the clustering contrib (reverts SOLR-4708). (Dawid Weiss)

        Show
        ASF subversion and git services added a comment - Commit 1531377 from Dawid Weiss in branch 'dev/trunk' [ https://svn.apache.org/r1531377 ] SOLR-5323 : Disable ClusteringComponent by default in collection1 example. The solr.clustering.enabled system property needs to be set to 'true' to enable the clustering contrib (reverts SOLR-4708 ). (Dawid Weiss)
        Hide
        ASF subversion and git services added a comment -

        Commit 1531378 from Dawid Weiss in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1531378 ]

        SOLR-5323: Disable ClusteringComponent by default in collection1 example. The solr.clustering.enabled system property needs to be set to 'true' to enable the clustering contrib (reverts SOLR-4708). (Dawid Weiss)

        Show
        ASF subversion and git services added a comment - Commit 1531378 from Dawid Weiss in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1531378 ] SOLR-5323 : Disable ClusteringComponent by default in collection1 example. The solr.clustering.enabled system property needs to be set to 'true' to enable the clustering contrib (reverts SOLR-4708 ). (Dawid Weiss)
        Hide
        ASF subversion and git services added a comment -

        Commit 1531380 from Dawid Weiss in branch 'dev/branches/lucene_solr_4_5'
        [ https://svn.apache.org/r1531380 ]

        SOLR-5323: Disable ClusteringComponent by default in collection1 example. The solr.clustering.enabled system property needs to be set to 'true' to enable the clustering contrib (reverts SOLR-4708). (Dawid Weiss)

        Show
        ASF subversion and git services added a comment - Commit 1531380 from Dawid Weiss in branch 'dev/branches/lucene_solr_4_5' [ https://svn.apache.org/r1531380 ] SOLR-5323 : Disable ClusteringComponent by default in collection1 example. The solr.clustering.enabled system property needs to be set to 'true' to enable the clustering contrib (reverts SOLR-4708 ). (Dawid Weiss)
        Hide
        Dawid Weiss added a comment -

        Applied to branch_4x, lucene_solr_4_5 and trunk.

        Show
        Dawid Weiss added a comment - Applied to branch_4x, lucene_solr_4_5 and trunk.

          People

          • Assignee:
            Dawid Weiss
            Reporter:
            John Berryman
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development