Solr
  1. Solr
  2. SOLR-7227

Solr distribution archive should have the WAR file extracted already

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.0
    • Fix Version/s: 5.3, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, there is still the solr.war file in the server/webapps directory, which gets extracted upon first startup of Solr. It would be better to ship Solr with the WAR already extracted, thus taking us one step closer to truly not shipping a WAR file. Moreover, some users have reported not being able to make /opt/solr truly read-only because of the need to extract the WAR on-the-fly upon first startup.

      1. SOLR-7227.patch
        17 kB
        Timothy Potter
      2. SOLR-7227.patch
        15 kB
        Timothy Potter
      3. SOLR-7227-part2.patch
        17 kB
        Uwe Schindler
      4. SOLR-7227-part2.patch
        12 kB
        Uwe Schindler
      5. SOLR-7227-part2.patch
        7 kB
        Uwe Schindler
      6. SOLR-7227-part2.patch
        7 kB
        Uwe Schindler
      7. SOLR-7227-part2.patch
        6 kB
        Uwe Schindler
      8. SOLR-7227-part2.patch
        4 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          Shawn Heisey added a comment -

          Would the plan here be to eliminate the war itself in the binary release and just include the extracted archive? I would hate to add another 30MB to the download size.

          Show
          Shawn Heisey added a comment - Would the plan here be to eliminate the war itself in the binary release and just include the extracted archive? I would hate to add another 30MB to the download size.
          Hide
          Shawn Heisey added a comment - - edited

          Very good upgrade instructions will be very important, as the old extracted archive (and any war files) must be completely deleted before copying the new version into place. This is good advice even when a war file is being used.

          Show
          Shawn Heisey added a comment - - edited Very good upgrade instructions will be very important, as the old extracted archive (and any war files) must be completely deleted before copying the new version into place. This is good advice even when a war file is being used.
          Hide
          Timothy Potter added a comment - - edited

          Would the plan here be to eliminate the war itself in the binary release and just include the extracted archive?

          Correct. No more WAR period!

          Very good upgrade instructions will be very important

          Doing this actually helps with upgrades since you can just install a new version into /opt, e.g. /opt/solr-5.1.0, and then update the symlink /opt/solr -> /opt/solr-5.1.0 ... assuming you're using the suggested production deployment model describe here: https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production

          Show
          Timothy Potter added a comment - - edited Would the plan here be to eliminate the war itself in the binary release and just include the extracted archive? Correct. No more WAR period! Very good upgrade instructions will be very important Doing this actually helps with upgrades since you can just install a new version into /opt, e.g. /opt/solr-5.1.0, and then update the symlink /opt/solr -> /opt/solr-5.1.0 ... assuming you're using the suggested production deployment model describe here: https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production
          Hide
          Shawn Heisey added a comment -

          and then update the symlink /opt/solr -> /opt/solr-5.1.0

          There are a lot of reasons (other than a strong dislike of Microsoft's general approach to software licensing) that I prefer Linux to Windows. This is one of them. NTFS does have similar functionality to links, but it is very much an expert option which is not at all well-known even among seasoned administrators.

          Show
          Shawn Heisey added a comment - and then update the symlink /opt/solr -> /opt/solr-5.1.0 There are a lot of reasons (other than a strong dislike of Microsoft's general approach to software licensing) that I prefer Linux to Windows. This is one of them. NTFS does have similar functionality to links, but it is very much an expert option which is not at all well-known even among seasoned administrators.
          Hide
          Timothy Potter added a comment -

          I'm porting a ton of code from bin/solr and bin/solr.cmd over to Java as part of the SOLR-7043 effort and have hit a problem on Windows where we can't launch the SolrCLI application until solr.war is extracted ... so it's time to get this one done for 5.3

          Show
          Timothy Potter added a comment - I'm porting a ton of code from bin/solr and bin/solr.cmd over to Java as part of the SOLR-7043 effort and have hit a problem on Windows where we can't launch the SolrCLI application until solr.war is extracted ... so it's time to get this one done for 5.3
          Hide
          Timothy Potter added a comment -

          Here's a patch that explodes the server/webapps/solr.war into server/solr-webapp/webapp and then removes solr.war.

          This gives us a couple of nice things:
          1) Jetty no longer has to extract the solr.war, so production installs can make the distribution folder read-only, i.e. /opt/solr can be read-only with all writing happening in /var/solr

          2) No more need to extract the WAR to run tools like zkCli.sh and SolrCLI (esp. problematic on Windows where you don't have JAR if you're running with a JRE instead of JDK)

          3) One step closer to truly no war

          4) Reduces the size of the tgz file by about 12MB

          nightly-smoke passes with this patch but I'd still appreciate a review of the smoke test python code I had to change

          Lastly, I'm wondering if we shouldn't delete the server/webapps directory too? I left it in there now, but I think it should go away. And while we're at it, maybe we should rename the server/solr-webapp directory to something non-webapp specific, server/solr-??? (i'm terrible at naming things but seems like a reasonable time to remove any mention of webapp)

          Show
          Timothy Potter added a comment - Here's a patch that explodes the server/webapps/solr.war into server/solr-webapp/webapp and then removes solr.war. This gives us a couple of nice things: 1) Jetty no longer has to extract the solr.war, so production installs can make the distribution folder read-only, i.e. /opt/solr can be read-only with all writing happening in /var/solr 2) No more need to extract the WAR to run tools like zkCli.sh and SolrCLI (esp. problematic on Windows where you don't have JAR if you're running with a JRE instead of JDK) 3) One step closer to truly no war 4) Reduces the size of the tgz file by about 12MB nightly-smoke passes with this patch but I'd still appreciate a review of the smoke test python code I had to change Lastly, I'm wondering if we shouldn't delete the server/webapps directory too? I left it in there now, but I think it should go away. And while we're at it, maybe we should rename the server/solr-webapp directory to something non-webapp specific, server/solr-??? (i'm terrible at naming things but seems like a reasonable time to remove any mention of webapp)
          Hide
          Timothy Potter added a comment -

          Updated patch that removes the server/webapps directory as well; there's no sense in shipping Solr with an empty webapps directory esp. since we're trying to dispel the idea that Solr is a Web application.

          This is ready for commit from where I sit.

          Show
          Timothy Potter added a comment - Updated patch that removes the server/webapps directory as well; there's no sense in shipping Solr with an empty webapps directory esp. since we're trying to dispel the idea that Solr is a Web application. This is ready for commit from where I sit.
          Hide
          ASF subversion and git services added a comment -

          Commit 1692925 from Timothy Potter in branch 'dev/trunk'
          [ https://svn.apache.org/r1692925 ]

          SOLR-7227: Solr distribution archive should have the WAR file extracted already

          Show
          ASF subversion and git services added a comment - Commit 1692925 from Timothy Potter in branch 'dev/trunk' [ https://svn.apache.org/r1692925 ] SOLR-7227 : Solr distribution archive should have the WAR file extracted already
          Hide
          ASF subversion and git services added a comment -

          Commit 1692947 from Timothy Potter in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1692947 ]

          SOLR-7227: Solr distribution archive should have the WAR file extracted already

          Show
          ASF subversion and git services added a comment - Commit 1692947 from Timothy Potter in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1692947 ] SOLR-7227 : Solr distribution archive should have the WAR file extracted already
          Hide
          Uwe Schindler added a comment -

          Hi,
          just one question: Why do we create the WAR file at all in the build.xml of webapps module? Currently we create it and delete it in the the same ANT target after extracting! I would simply replace the <war/> by a <copy/> task. Some attributes like the path to the web.xml need to be changed (as the copy task does not have them, but otherwise the ANT War task does nothing special. The manifest is useless.

          Show
          Uwe Schindler added a comment - Hi, just one question: Why do we create the WAR file at all in the build.xml of webapps module? Currently we create it and delete it in the the same ANT target after extracting! I would simply replace the <war/> by a <copy/> task. Some attributes like the path to the web.xml need to be changed (as the copy task does not have them, but otherwise the ANT War task does nothing special. The manifest is useless.
          Hide
          Uwe Schindler added a comment - - edited

          Patch removing webapp WAR build completely. I also renamed some targets.

          Show
          Uwe Schindler added a comment - - edited Patch removing webapp WAR build completely. I also renamed some targets.
          Hide
          Uwe Schindler added a comment -

          Removal of more useless stuff

          Show
          Uwe Schindler added a comment - Removal of more useless stuff
          Hide
          Uwe Schindler added a comment -

          I accidently removed additional webapp folder, reverted.

          Show
          Uwe Schindler added a comment - I accidently removed additional webapp folder, reverted.
          Hide
          Uwe Schindler added a comment -

          One more obsolete property. I also added "extractWAR"="false" to the context descriptor: this prevents Jetty from creating the temporary folder, that leaves checkout dirty. This helps with read-only filesystems, too.

          Seems ready to commit now.

          I noticed one thing: the inner folder solr-webapp/webapp is somehow obsolete, we could move the whole stuff one up (remove inner webapp folder). This should maybe a separate issue/commit, because this affects script files, too.

          Show
          Uwe Schindler added a comment - One more obsolete property. I also added "extractWAR"="false" to the context descriptor: this prevents Jetty from creating the temporary folder, that leaves checkout dirty. This helps with read-only filesystems, too. Seems ready to commit now. I noticed one thing: the inner folder solr-webapp/webapp is somehow obsolete, we could move the whole stuff one up (remove inner webapp folder). This should maybe a separate issue/commit, because this affects script files, too.
          Hide
          Uwe Schindler added a comment -

          By the way: I compared the pre-patch and after-patch extracted webapp folders. They are identical, only the META-INF folder is missing (which is fine, because its just a packaging relict).

          Show
          Uwe Schindler added a comment - By the way: I compared the pre-patch and after-patch extracted webapp folders. They are identical, only the META-INF folder is missing (which is fine, because its just a packaging relict).
          Hide
          Timothy Potter added a comment -

          I'm pretty sure the smoke tester checks things in the manifest. Uwe Schindler Did you run smoke tester with your patch?

          Show
          Timothy Potter added a comment - I'm pretty sure the smoke tester checks things in the manifest. Uwe Schindler Did you run smoke tester with your patch?
          Hide
          Uwe Schindler added a comment -

          Yes passes. Manifests are only required inside JAR or WAR files (they have metadata about the WAR file itsself) - and the WAR file is gone. The JAR files of our application all have valid META-INF.

          Show
          Uwe Schindler added a comment - Yes passes. Manifests are only required inside JAR or WAR files (they have metadata about the WAR file itsself) - and the WAR file is gone. The JAR files of our application all have valid META-INF.
          Hide
          Uwe Schindler added a comment -

          But in any case we can remove the special cases for WAR files from the smoke tester, but this is why you left the issue open.

          Show
          Uwe Schindler added a comment - But in any case we can remove the special cases for WAR files from the smoke tester, but this is why you left the issue open.
          Hide
          Uwe Schindler added a comment - - edited

          New patch without WAR special case in smoke tester (no longer needed) – currently untested, have to boot Linux again.

          Show
          Uwe Schindler added a comment - - edited New patch without WAR special case in smoke tester (no longer needed) – currently untested, have to boot Linux again.
          Hide
          Timothy Potter added a comment -

          thanks - I'm running it now too on my Mac

          Show
          Timothy Potter added a comment - thanks - I'm running it now too on my Mac
          Hide
          Uwe Schindler added a comment -

          Do you use the latest patch? The old one may not yet have that WAR file parts removed. The old smoker worked on cygwin, but you never know

          Show
          Uwe Schindler added a comment - Do you use the latest patch? The old one may not yet have that WAR file parts removed. The old smoker worked on cygwin, but you never know
          Hide
          Uwe Schindler added a comment - - edited

          Otherwise do you have an opinion about the extra inner folder solr-webapp/webapp? We should remove the inner webapp folder, I just did not do this, because I have no idea which scripts are affected by this. I wanted to look into that later or in a separate commit. I just wanted to get rid of the WAR completely.

          Show
          Uwe Schindler added a comment - - edited Otherwise do you have an opinion about the extra inner folder solr-webapp/webapp? We should remove the inner webapp folder, I just did not do this, because I have no idea which scripts are affected by this. I wanted to look into that later or in a separate commit. I just wanted to get rid of the WAR completely.
          Hide
          Timothy Potter added a comment -

          yes, using latest patch you posted at 14:52 ...

          We might as well address the extra folder now too ... the scripts affected are minimal (smoketester, bin/solr, bin/solr.cmd, zkcli.sh/cmd, and a few in the cloud-dev).

          Show
          Timothy Potter added a comment - yes, using latest patch you posted at 14:52 ... We might as well address the extra folder now too ... the scripts affected are minimal (smoketester, bin/solr, bin/solr.cmd, zkcli.sh/cmd, and a few in the cloud-dev).
          Hide
          Shawn Heisey added a comment -

          Otherwise do you have an opinion about the extra inner folder solr-webapp/webapp?

          I'm not sure we should move the artifacts out of the inner webapp directory, at least not until the next major version.

          My concern is not our own scripts. We can change those easily enough. The potential problem is homegrown scripts written by users. If we move the extracted artifacts, even just one directory level, we risk problems with highly customized user setups.

          Is it enough to assume someone who builds their own scripts will be able to use a note in the "upgrading from" section of CHANGES.txt to figure out how to fix their setup when they upgrade? It might be. User confusion is always a worry for me, because Solr already has plenty to offer in that department.

          I can't imagine based on anything you've said that I would want to vote -1. I offer my thoughts only for consideration.

          Semi-related: I need to find out what we've got for documentation on upgrading a Solr 5.x install to the next release. I have some ideas about how I would do it, but I'd like to know what (if anything) we are saying officially.

          Show
          Shawn Heisey added a comment - Otherwise do you have an opinion about the extra inner folder solr-webapp/webapp? I'm not sure we should move the artifacts out of the inner webapp directory, at least not until the next major version. My concern is not our own scripts. We can change those easily enough. The potential problem is homegrown scripts written by users. If we move the extracted artifacts, even just one directory level, we risk problems with highly customized user setups. Is it enough to assume someone who builds their own scripts will be able to use a note in the "upgrading from" section of CHANGES.txt to figure out how to fix their setup when they upgrade? It might be. User confusion is always a worry for me, because Solr already has plenty to offer in that department. I can't imagine based on anything you've said that I would want to vote -1. I offer my thoughts only for consideration. Semi-related: I need to find out what we've got for documentation on upgrading a Solr 5.x install to the next release. I have some ideas about how I would do it, but I'd like to know what (if anything) we are saying officially.
          Hide
          Uwe Schindler added a comment -

          Smoke tester passes for me on Linux. There is an unrelated bug in smoker when it tries to execute post.jar (it does not setup PATH correctly as its does for the scripts), so it fails if you have no "java" in your path, or it executes the wrong Java (maybe older version). I will open separate issue, its really unrelated. It just costed me half an hour

          Shawn: I have no strong opinion, we can leave it as it is. But custom scripts may already break because there is no WAR anymore. In previous versions, Jetty extracted the WAR to a temporary folder, so the scripts will for sure no longer work.

          Show
          Uwe Schindler added a comment - Smoke tester passes for me on Linux. There is an unrelated bug in smoker when it tries to execute post.jar (it does not setup PATH correctly as its does for the scripts), so it fails if you have no "java" in your path, or it executes the wrong Java (maybe older version). I will open separate issue, its really unrelated. It just costed me half an hour Shawn: I have no strong opinion, we can leave it as it is. But custom scripts may already break because there is no WAR anymore. In previous versions, Jetty extracted the WAR to a temporary folder, so the scripts will for sure no longer work.
          Hide
          Shawn Heisey added a comment -

          But custom scripts may already break because there is no WAR anymore.

          Very true, and that is something that I had not considered. I also have no strong opinion, and it sounds like this entire change is destined to lead to user confusion, so let's jump in all the way!

          Show
          Shawn Heisey added a comment - But custom scripts may already break because there is no WAR anymore. Very true, and that is something that I had not considered. I also have no strong opinion, and it sounds like this entire change is destined to lead to user confusion, so let's jump in all the way!
          Hide
          Uwe Schindler added a comment -

          I reverted the Smoker changes from the previous commit and removed the WAR stuff completely. This is now much cleaner,

          Show
          Uwe Schindler added a comment - I reverted the Smoker changes from the previous commit and removed the WAR stuff completely. This is now much cleaner,
          Hide
          Uwe Schindler added a comment -

          I would like to commit the current patch and then later fix the extra folder. Timothy could also handle that separately. This patch is just to fix the useless WAR file create and reverts the META-INF checks.

          Is it OK to commit?

          Show
          Uwe Schindler added a comment - I would like to commit the current patch and then later fix the extra folder. Timothy could also handle that separately. This patch is just to fix the useless WAR file create and reverts the META-INF checks. Is it OK to commit?
          Hide
          Timothy Potter added a comment -

          +1 to commit as is

          Show
          Timothy Potter added a comment - +1 to commit as is
          Hide
          ASF subversion and git services added a comment -

          Commit 1693143 from Uwe Schindler in branch 'dev/trunk'
          [ https://svn.apache.org/r1693143 ]

          SOLR-7227: Don't create the WAR file at all

          Show
          ASF subversion and git services added a comment - Commit 1693143 from Uwe Schindler in branch 'dev/trunk' [ https://svn.apache.org/r1693143 ] SOLR-7227 : Don't create the WAR file at all
          Hide
          ASF subversion and git services added a comment -

          Commit 1693144 from Uwe Schindler in branch 'dev/trunk'
          [ https://svn.apache.org/r1693144 ]

          SOLR-7227: Be sure to not put servlet.jar into webapp

          Show
          ASF subversion and git services added a comment - Commit 1693144 from Uwe Schindler in branch 'dev/trunk' [ https://svn.apache.org/r1693144 ] SOLR-7227 : Be sure to not put servlet.jar into webapp
          Hide
          ASF subversion and git services added a comment -

          Commit 1693145 from Uwe Schindler in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1693145 ]

          Merged revision(s) 1693143,1693144 from lucene/dev/trunk:
          SOLR-7227: Don't create the WAR file at all

          Show
          ASF subversion and git services added a comment - Commit 1693145 from Uwe Schindler in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1693145 ] Merged revision(s) 1693143,1693144 from lucene/dev/trunk: SOLR-7227 : Don't create the WAR file at all
          Hide
          Uwe Schindler added a comment -

          OK, committed. The remaining stuff is now the extra folder. Smoke tester should be fine.

          Show
          Uwe Schindler added a comment - OK, committed. The remaining stuff is now the extra folder. Smoke tester should be fine.
          Hide
          Timothy Potter added a comment -

          This was fixed in 5.3, just didn't get closed out. Thanks for the help on this Uwe

          Show
          Timothy Potter added a comment - This was fixed in 5.3, just didn't get closed out. Thanks for the help on this Uwe

            People

            • Assignee:
              Timothy Potter
              Reporter:
              Timothy Potter
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development