Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9109

Add support for custom ivysettings.xml

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.1
    • Component/s: None
    • Labels:
      None

      Description

      Currently solr/lucene/common-build.xml hardcodes file ivy-settings.xml in the ivy.configure task. It means that, unlike all other CDH components that use Ant, Solr does not allow the user to provide a custom ivysettings.xml.

      In the Cauldron CDH build, we need to make some adjustments in the build process to satisfy CDH-internal build dependencies with artifacts generated locally, rather than download them from repo1.maven.org etc. E.g. all component should use locally-generated avro-snapshot jars instead of publicly released ones, etc. For Ant, we achieve that by giving it a special ivysettings.xml file, that limits artifact downloading to the local on-disk maven repository and Cloudera artifactory server.

      All CDH components except Solr allow the user to specify a custom ivysettings.xml file by overriding -Divysettings.xml property. We need to add the same feature to Solr. It can be easily achieved by changing several lines in solr/lucene/common-build.xml.

      1. 0001-Support-for-custom-ivysettings.xml.patch
        2 kB
        Misha Dmitriev
      2. SOLR-9109.patch
        10 kB
        Steve Rowe
      3. SOLR-9109.patch
        9 kB
        Steve Rowe
      4. SOLR-9109.patch
        8 kB
        Steve Rowe
      5. SOLR-9109.patch
        8 kB
        Steve Rowe
      6. SOLR-9109.patch
        0.0 kB
        Misha Dmitriev

        Issue Links

          Activity

          Hide
          dancollins Daniel Collins added a comment -

          +1 for this for a different reason. We build Solr in a corporate environment which means we are behind firewalls, and so can't use the public repos. Currently we have to patch Solr in order to build it, which is not ideal. If we had a custom extra file, that would be fine, but patching an existing Solr file doesn't feel right.

          Show
          dancollins Daniel Collins added a comment - +1 for this for a different reason. We build Solr in a corporate environment which means we are behind firewalls, and so can't use the public repos. Currently we have to patch Solr in order to build it, which is not ideal. If we had a custom extra file, that would be fine, but patching an existing Solr file doesn't feel right.
          Hide
          steve_rowe Steve Rowe added a comment -

          Is this addition necessary?:

          +  <property file="${common.dir}/ivy-versions.properties" />
          +
          
          Show
          steve_rowe Steve Rowe added a comment - Is this addition necessary?: + <property file="${common.dir}/ivy-versions.properties" /> +
          Hide
          misha@cloudera.com Misha Dmitriev added a comment -

          It is. It defines several properties such as $

          {ant.version}

          IIRC, which are used in names of artifacts to download. Before this change, ivy-versions.properties was processed only in the Solr's hardcoded ivy-settings.xml.

          Show
          misha@cloudera.com Misha Dmitriev added a comment - It is. It defines several properties such as $ {ant.version} IIRC, which are used in names of artifacts to download. Before this change, ivy-versions.properties was processed only in the Solr's hardcoded ivy-settings.xml.
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          Please don't pollute the global Ant properties with those settings, especially as they have no namespacing prefix! So the user should then add it to his own Ivy.xml file if he wants to exchange it.

          If we want this, remove the include of the properties file from some componenets in Ant that use it (we include it in private scopes for some tasks),and please add a unique prefix to all properties in the file.

          Show
          thetaphi Uwe Schindler added a comment - - edited Please don't pollute the global Ant properties with those settings, especially as they have no namespacing prefix! So the user should then add it to his own Ivy.xml file if he wants to exchange it. If we want this, remove the include of the properties file from some componenets in Ant that use it (we include it in private scopes for some tasks),and please add a unique prefix to all properties in the file.
          Hide
          misha@cloudera.com Misha Dmitriev added a comment -

          I am not familiar with fine-level details of how Ant works, so let's make sure that we are on the same page and clarify what happens here. Currently (before my change) there is the following relationship between files under solr/lucene (arrow means "sources" or "includes"):

          common-build.xml -> ivy-settings.xml -> ivy-versions.properties

          after my change:

          common-build.xml -> ivy-versions.properties
          common-build.xml -> ivy-settings.xml (or another ivysettings file if supplied/overridden)

          So, does the visibility scope of properties from ivy-versions.properties (that's what I understand you are concerned about) change after my modification? Even if yes, does it matter given that ivy-versions.properties belongs to solr/lucene and I don't change anything in it?

          Show
          misha@cloudera.com Misha Dmitriev added a comment - I am not familiar with fine-level details of how Ant works, so let's make sure that we are on the same page and clarify what happens here. Currently (before my change) there is the following relationship between files under solr/lucene (arrow means "sources" or "includes"): common-build.xml -> ivy-settings.xml -> ivy-versions.properties after my change: common-build.xml -> ivy-versions.properties common-build.xml -> ivy-settings.xml (or another ivysettings file if supplied/overridden) So, does the visibility scope of properties from ivy-versions.properties (that's what I understand you are concerned about) change after my modification? Even if yes, does it matter given that ivy-versions.properties belongs to solr/lucene and I don't change anything in it?
          Hide
          steve_rowe Steve Rowe added a comment -

          IIRC, which are used in names of artifacts to download. Before this change, ivy-versions.properties was processed only in the Solr's hardcoded ivy-settings.xml.

          ivy-settings.xml is Lucene's, not Solr's.

          Why can't user-supplied ivy-settings.xml files include ivy-versions.properties in the same way that the OOTB ivy-settings.xml does?

          Show
          steve_rowe Steve Rowe added a comment - IIRC, which are used in names of artifacts to download. Before this change, ivy-versions.properties was processed only in the Solr's hardcoded ivy-settings.xml. ivy-settings.xml is Lucene's, not Solr's. Why can't user-supplied ivy-settings.xml files include ivy-versions.properties in the same way that the OOTB ivy-settings.xml does?
          Hide
          misha@cloudera.com Misha Dmitriev added a comment -

          Regarding Lucene vs Solr: please note that both before and after my change all the changes are confined to Lucene's common-build.xml file. It would be good to understand whether my change makes any difference wrt. the visibility of properties defined in ivy-versions.properties. Perhaps an experiment demonstrating that after the change the properties propagate somewhere they are not supposed to be visible?

          As for including ivy-versions.properties in the new user-supplied ivysettings.xml. The whole point of making my change is to make all CDH components use the same single ivysettings.xml file, that just defines several supported locations to download artifacts from. Adding Solr-specific stuff to it may cause problems for other components. We can, in the worst case, play some tricks. Like, create at run time a customized copy of our ivysettings.xml with ivy-versions.properties "injected" into it. But this is obviously not pretty. So it would be good to understand the exact reason why we need such a special effort.

          Show
          misha@cloudera.com Misha Dmitriev added a comment - Regarding Lucene vs Solr: please note that both before and after my change all the changes are confined to Lucene's common-build.xml file. It would be good to understand whether my change makes any difference wrt. the visibility of properties defined in ivy-versions.properties. Perhaps an experiment demonstrating that after the change the properties propagate somewhere they are not supposed to be visible? As for including ivy-versions.properties in the new user-supplied ivysettings.xml. The whole point of making my change is to make all CDH components use the same single ivysettings.xml file, that just defines several supported locations to download artifacts from. Adding Solr-specific stuff to it may cause problems for other components. We can, in the worst case, play some tricks. Like, create at run time a customized copy of our ivysettings.xml with ivy-versions.properties "injected" into it. But this is obviously not pretty. So it would be good to understand the exact reason why we need such a special effort.
          Hide
          steve_rowe Steve Rowe added a comment -

          Note that the <properties> tag in ivy-settings.xml populates Ivy variables, not Ant properties, so there is no property leakage outside of the scope in which those definitions are required.

          This patch should enable what you want while still retaining the exact same scope as previously, by using nested ivy-settings.xml files.

          Show
          steve_rowe Steve Rowe added a comment - Note that the <properties> tag in ivy-settings.xml populates Ivy variables, not Ant properties, so there is no property leakage outside of the scope in which those definitions are required. This patch should enable what you want while still retaining the exact same scope as previously, by using nested ivy-settings.xml files.
          Hide
          steve_rowe Steve Rowe added a comment - - edited

          Sorry, previous patch didn't work because I changed a file name without changing where it was referred from.

          Using this version of the patch, ant clean-jars resolve works for me.

          I also successfully specified an alternative ivy settings file via ant -Divysettings.xml=/path/to/my-ivy-settings.xml clean-jars resolve (after first removing the default nested ivy settings file) - note that I provided a full path there, because the relative path is interpreted relative to the lucene/ subdir, not the top-level directory.

          Show
          steve_rowe Steve Rowe added a comment - - edited Sorry, previous patch didn't work because I changed a file name without changing where it was referred from. Using this version of the patch, ant clean-jars resolve works for me. I also successfully specified an alternative ivy settings file via ant -Divysettings.xml=/path/to/my-ivy-settings.xml clean-jars resolve (after first removing the default nested ivy settings file) - note that I provided a full path there, because the relative path is interpreted relative to the lucene/ subdir, not the top-level directory.
          Hide
          misha@cloudera.com Misha Dmitriev added a comment -

          Thank you very much, Steve! I appreciate your help a lot.

          Just to be 100% sure, I'll run the build to verify that your patch works for us as expected.

          Assuming it does - what will be the next steps?

          Show
          misha@cloudera.com Misha Dmitriev added a comment - Thank you very much, Steve! I appreciate your help a lot. Just to be 100% sure, I'll run the build to verify that your patch works for us as expected. Assuming it does - what will be the next steps?
          Hide
          steve_rowe Steve Rowe added a comment -

          If we want this, remove the include of the properties file from some componenets in Ant that use it (we include it in private scopes for some tasks),and please add a unique prefix to all properties in the file.

          Uwe Schindler, there is one place where this is not done: <target name="-check-forbidden-all"> in solr/common-build.xml:

              <property file="${common.dir}/ivy-versions.properties"/> <!-- for commons-io version -->
          
          Show
          steve_rowe Steve Rowe added a comment - If we want this, remove the include of the properties file from some componenets in Ant that use it (we include it in private scopes for some tasks),and please add a unique prefix to all properties in the file. Uwe Schindler , there is one place where this is not done: <target name="-check-forbidden-all"> in solr/common-build.xml : <property file= "${common.dir}/ivy-versions.properties" /> <!-- for commons-io version -->
          Hide
          steve_rowe Steve Rowe added a comment -

          Thank you very much, Steve! I appreciate your help a lot.
          Just to be 100% sure, I'll run the build to verify that your patch works for us as expected.
          Assuming it does - what will be the next steps?

          If it works for you, and if there are no objections (hoping Uwe will +1), I'll commit it, and the change will be included in the 6.1 release.

          Show
          steve_rowe Steve Rowe added a comment - Thank you very much, Steve! I appreciate your help a lot. Just to be 100% sure, I'll run the build to verify that your patch works for us as expected. Assuming it does - what will be the next steps? If it works for you, and if there are no objections (hoping Uwe will +1), I'll commit it, and the change will be included in the 6.1 release.
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          Hi Steve,

          yes that trick works. I was thinking about the same last hour, but was not sure if this works. But yes, it works, because Ant propertes can be seen by ivy, but not the other way round.

          About the solr common-build.xml file and forbidden: I thought the include was private to the target, but you are right, its global and polutes. There is another one using the ivy-versions.properties, where it includes a prefix: See lucene/analyzers/common/build.xml for getting icu4j version:

            <target xmlns:ivy="antlib:org.apache.ivy.ant" name="-resolve-icu4j" unless="icu4j.resolved" depends="ivy-availability-check,ivy-configure">
              <loadproperties prefix="ivyversions" srcFile="${common.dir}/ivy-versions.properties"/>
              <ivy:cachepath organisation="com.ibm.icu" module="icu4j" revision="${ivyversions./com.ibm.icu/icu4j}"
                inline="true" conf="default" transitive="true" pathid="icu4j.classpath"/>
              <property name="icu4j.resolved" value="true"/>
            </target>
          

          Can we change the forbiddenapis one to work in a similar way? I can do that in a separate commit, but as you are already working on it, maybe change it!

          Show
          thetaphi Uwe Schindler added a comment - - edited Hi Steve, yes that trick works. I was thinking about the same last hour, but was not sure if this works. But yes, it works, because Ant propertes can be seen by ivy, but not the other way round. About the solr common-build.xml file and forbidden: I thought the include was private to the target, but you are right, its global and polutes. There is another one using the ivy-versions.properties, where it includes a prefix: See lucene/analyzers/common/build.xml for getting icu4j version: <target xmlns:ivy = "antlib:org.apache.ivy.ant" name= "-resolve-icu4j" unless= "icu4j.resolved" depends= "ivy-availability-check,ivy-configure" > <loadproperties prefix= "ivyversions" srcFile= "${common.dir}/ivy-versions.properties" /> <ivy:cachepath organisation= "com.ibm.icu" module= "icu4j" revision= "${ivyversions./com.ibm.icu/icu4j}" inline= "true" conf= "default" transitive= "true" pathid= "icu4j.classpath" /> <property name= "icu4j.resolved" value= "true" /> </target> Can we change the forbiddenapis one to work in a similar way? I can do that in a separate commit, but as you are already working on it, maybe change it!
          Hide
          steve_rowe Steve Rowe added a comment -

          Can we change the forbiddenapis one to work in a similar way? I can do that in a separate commit, but as you are already working on it, maybe change it!

          Sure, this version of the patch adds a "ivyversions." prefix to the properties loaded in that target.

          Show
          steve_rowe Steve Rowe added a comment - Can we change the forbiddenapis one to work in a similar way? I can do that in a separate commit, but as you are already working on it, maybe change it! Sure, this version of the patch adds a "ivyversions." prefix to the properties loaded in that target.
          Hide
          thetaphi Uwe Schindler added a comment -

          LGTM! +1
          (thanks for fixing the forbiddenapis task)

          Show
          thetaphi Uwe Schindler added a comment - LGTM! +1 (thanks for fixing the forbiddenapis task)
          Hide
          misha@cloudera.com Misha Dmitriev added a comment -

          I've just verified that Steve's patch works as expected in the Cauldron build. Thank you very much again, and please go ahead and commit it.

          When is Solr 6.1 expected to be released?

          Show
          misha@cloudera.com Misha Dmitriev added a comment - I've just verified that Steve's patch works as expected in the Cauldron build. Thank you very much again, and please go ahead and commit it. When is Solr 6.1 expected to be released?
          Hide
          steve_rowe Steve Rowe added a comment -

          Okay, I'll commit it - here's the final patch with a CHANGES.txt entry and some comments in the Ivy settings files.

          I expect 6.1 will be released in roughly a month. Once the process starts, which it has yet to do for 6.1, it takes 2-4 weeks for a release to be put together, vetted, and published, assuming no major problems are found.

          Show
          steve_rowe Steve Rowe added a comment - Okay, I'll commit it - here's the final patch with a CHANGES.txt entry and some comments in the Ivy settings files. I expect 6.1 will be released in roughly a month. Once the process starts, which it has yet to do for 6.1, it takes 2-4 weeks for a release to be put together, vetted, and published, assuming no major problems are found.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 82788504aa3125075afc606413c0603a86cd4763 in lucene-solr's branch refs/heads/master from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8278850 ]

          SOLR-9109: Allow specification of a custom Ivy settings file via system property "ivysettings.xml".

          Show
          jira-bot ASF subversion and git services added a comment - Commit 82788504aa3125075afc606413c0603a86cd4763 in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8278850 ] SOLR-9109 : Allow specification of a custom Ivy settings file via system property "ivysettings.xml".
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0ec57a9918761cb5afd9b043e56299410da1d989 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0ec57a9 ]

          SOLR-9109: Allow specification of a custom Ivy settings file via system property "ivysettings.xml".

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0ec57a9918761cb5afd9b043e56299410da1d989 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0ec57a9 ] SOLR-9109 : Allow specification of a custom Ivy settings file via system property "ivysettings.xml".
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a5be64af055039431770a93026792b52b3389585 in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a5be64a ]

          SOLR-9109: the $

          {settings.xml}

          is a file path, not a URL

          Show
          jira-bot ASF subversion and git services added a comment - Commit a5be64af055039431770a93026792b52b3389585 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a5be64a ] SOLR-9109 : the $ {settings.xml} is a file path, not a URL
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 8df8243f536dcfe5372ed1125e9813a343ce34c1 in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8df8243 ]

          SOLR-9109: the $

          {settings.xml}

          is a file path, not a URL

          Show
          jira-bot ASF subversion and git services added a comment - Commit 8df8243f536dcfe5372ed1125e9813a343ce34c1 in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8df8243 ] SOLR-9109 : the $ {settings.xml} is a file path, not a URL
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e545c696c3b943146e0be3bf317382f12436186e in lucene-solr's branch refs/heads/master from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e545c69 ]

          SOLR-9109: tell the smoke tester and the check-lib-versions target about the renamed Ivy settings files

          Show
          jira-bot ASF subversion and git services added a comment - Commit e545c696c3b943146e0be3bf317382f12436186e in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e545c69 ] SOLR-9109 : tell the smoke tester and the check-lib-versions target about the renamed Ivy settings files
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 2dbc164072e59e319caf009873a5c218418421c4 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2dbc164 ]

          SOLR-9109: tell the smoke tester and the check-lib-versions target about the renamed Ivy settings files

          Show
          jira-bot ASF subversion and git services added a comment - Commit 2dbc164072e59e319caf009873a5c218418421c4 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2dbc164 ] SOLR-9109 : tell the smoke tester and the check-lib-versions target about the renamed Ivy settings files
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5fb11323b6a225d2d88984aa024d2fd7eec8bf24 in lucene-solr's branch refs/heads/master from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5fb1132 ]

          SOLR-9109: add missing comma in smokeTestRelease.py extras list

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5fb11323b6a225d2d88984aa024d2fd7eec8bf24 in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5fb1132 ] SOLR-9109 : add missing comma in smokeTestRelease.py extras list
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f73997bb412060f08d6657dfce6ceb8d0c0410eb in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f73997b ]

          SOLR-9109: add missing comma in smokeTestRelease.py extras list

          Show
          jira-bot ASF subversion and git services added a comment - Commit f73997bb412060f08d6657dfce6ceb8d0c0410eb in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f73997b ] SOLR-9109 : add missing comma in smokeTestRelease.py extras list
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Should we close this Steve Rowe?

          Show
          markrmiller@gmail.com Mark Miller added a comment - Should we close this Steve Rowe ?
          Hide
          steve_rowe Steve Rowe added a comment -

          Thanks for the reminder Mark Miller, I've resolved it.

          Show
          steve_rowe Steve Rowe added a comment - Thanks for the reminder Mark Miller , I've resolved it.

            People

            • Assignee:
              steve_rowe Steve Rowe
              Reporter:
              misha@cloudera.com Misha Dmitriev
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development