Solr
  1. Solr
  2. SOLR-118

Some admin pages stop working with "error 404" as the only symptom

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: web gui
    • Labels:
      None
    • Environment:

      Fedora Core 4 (Linux version 2.6.11-1.1369_FC4smp) Sun's JVM 1.5.0_07-b03

      Description

      This was reported to the mailing list a while ago, see http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200610.mbox/%3cf752f5320610311359y365d6322lb9b4e409f5557dfc@mail.gmail.com%3e

      Today I'm seeing the same thing on a Solr instance that has been running since January 9th (about 13 days) with the plain start.jar setup. Index contains 150'000 docs, 88322 search requests to date.

      $ curl http://localhost:8983/solr/admin/analysis.jsp
      <html>
      <head>
      <title>Error 404 /admin/analysis.jsp</title>
      </head>
      <body>
      <h2>HTTP ERROR: 404</h2><pre>/admin/analysis.jsp</pre>
      <p>RequestURI=/solr/admin/analysis.jsp</p>
      ...

      curl http://localhost:8983/solr/admin/index.jsp
      <html>
      <head>
      <title>Error 404 /admin/index.jsp</title>
      </head>
      <body>
      <h2>HTTP ERROR: 404</h2><pre>/admin/index.jsp</pre>
      <p>RequestURI=/solr/admin/index.jsp</p>
      ...

      Other admin pages work correctly, for example http://localhost:8983/solr/admin/stats.jsp

      I don't see any messages in the logs, which are capturing stdout and stderr from the JVM.

      I guess I'll have to restart this instance, I'm out of possibilities to find out what's happening exactly.

        Issue Links

          Activity

          Hoss Man made changes -
          Link This issue supercedes SOLR-2586 [ SOLR-2586 ]
          Bertrand Delacretaz made changes -
          Field Original Value New Value
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Bertrand Delacretaz added a comment -

          We have confirmed Otis's hypothesis by deleting Jetty's work dir under /tmp while Solr runs, which causes the 404 to appear as described above.

          I have added this to the Solr FAQ [1], and added Otis to my "IOU a glass of milk" list.

          [1] http://wiki.apache.org/solr/FAQ#head-eff0fcf7bd2ef9385999b1b5c71fffb34e568e8b

          Show
          Bertrand Delacretaz added a comment - We have confirmed Otis's hypothesis by deleting Jetty's work dir under /tmp while Solr runs, which causes the 404 to appear as described above. I have added this to the Solr FAQ [1] , and added Otis to my "IOU a glass of milk" list. [1] http://wiki.apache.org/solr/FAQ#head-eff0fcf7bd2ef9385999b1b5c71fffb34e568e8b
          Hide
          Otis Gospodnetic added a comment -

          Ah, yes, that will work. I have Jetty set up like that for http://simpy.com/ , that is why I knew the answer.

          Beverage of choice....hm, a glass of hot milk?

          Show
          Otis Gospodnetic added a comment - Ah, yes, that will work. I have Jetty set up like that for http://simpy.com/ , that is why I knew the answer. Beverage of choice....hm, a glass of hot milk?
          Hide
          Bertrand Delacretaz added a comment -

          Otis, I guess we'll owe you a free beverage of his choice, your guess sounds totally right.

          According to http://docs.codehaus.org/display/JETTY/Temporary+Directories the easiest fix might be to create a $(jetty.home)/work directory, which Jetty will use. I haven't checked if this works with the Jetty that's embedded with Solr.

          Show
          Bertrand Delacretaz added a comment - Otis, I guess we'll owe you a free beverage of his choice, your guess sounds totally right. According to http://docs.codehaus.org/display/JETTY/Temporary+Directories the easiest fix might be to create a $(jetty.home)/work directory, which Jetty will use. I haven't checked if this works with the Jetty that's embedded with Solr.
          Hide
          Otis Gospodnetic added a comment -

          Scanning the past comments....

          My first guess would be that Jetty was left configured as is by default, which means it expanded webapps in /tmp/Jetty__some_dir_here.... Some UNIX boxes are configured to purge old files from /tmp/ in order not tu run out of free disk space in /tmp partition. Perhaps this is what happened here.

          The fix is to configure Jetty to expand webapps to a dir that does not get purged. Of course, I cannot remember the property name for that.

          Show
          Otis Gospodnetic added a comment - Scanning the past comments.... My first guess would be that Jetty was left configured as is by default, which means it expanded webapps in /tmp/Jetty__some_dir_here.... Some UNIX boxes are configured to purge old files from /tmp/ in order not tu run out of free disk space in /tmp partition. Perhaps this is what happened here. The fix is to configure Jetty to expand webapps to a dir that does not get purged. Of course, I cannot remember the property name for that.
          Hide
          Hoss Man added a comment -

          i haven't reported anything, and i didn't really dig into it with much depth ... i just randomly discovered it while testing out something else. (This is the first time i've ever been able to reproduce the problem, i suspect because i tend to leave myself logged into my machines for years at a time with dozens of terminals open each dedicated to one and only one purpose ... so i'd never tried running the solr example code in more then one terminal – let alone the only two at the same time. the only reason i noticed today was because i'd brought up a new terminal last week to use for giving a solr demo on my second monitor (the first new terminal i've opened in about 2 months mind you) and i never shut it down until i noticed today that my regular solr example running terminal was misbehaving.)

          Show
          Hoss Man added a comment - i haven't reported anything, and i didn't really dig into it with much depth ... i just randomly discovered it while testing out something else. (This is the first time i've ever been able to reproduce the problem, i suspect because i tend to leave myself logged into my machines for years at a time with dozens of terminals open each dedicated to one and only one purpose ... so i'd never tried running the solr example code in more then one terminal – let alone the only two at the same time. the only reason i noticed today was because i'd brought up a new terminal last week to use for giving a solr demo on my second monitor (the first new terminal i've opened in about 2 months mind you) and i never shut it down until i noticed today that my regular solr example running terminal was misbehaving.)
          Hide
          Yonik Seeley added a comment -

          Nice find! Has this been reported to the Jetty folks, or is this something we can solve with configuration?

          Show
          Yonik Seeley added a comment - Nice find! Has this been reported to the Jetty folks, or is this something we can solve with configuration?
          Hide
          Hoss Man added a comment -

          FYI: I think i may have found one possible cause of this problem .. not sure if there are other causes.

          1) open two terminals both of which are in the solr example directory.
          2) in terminal A, run 'java -jar start.jar'
          3) note that the following URLs all work fine...
          http://localhost:8983/solr/
          http://localhost:8983/solr/select/?q=solr&version=2.2&start=0&rows=10&indent=on
          http://localhost:8983/solr/admin/
          4) in terminal B, run "java -jar start.jar"
          5) note that in terminal b, the server logs a "java.net.BindException: Address already in use" exception but the process does not terminate itself.
          6) note that while the process in terminal B is still running, the urls from step#3 still work fine, and the queries are logged to terminal A
          7) hit Ctrl-C in terminal B to kill that process, wait for it to log "Shutdown hook complete" and return your prompt
          8) note that the following URLs now 404...
          http://localhost:8983/solr/admin/
          http://localhost:8983/solr/
          9) note that this URL (and all non JSP non flat file solr urls) continues to work, and log the request to terminal A...
          http://localhost:8983/solr/select/?q=solr&version=2.2&start=0&rows=10&indent=on

          ..in short, i suspect this is a jetty problem, relating to people inadvertently trying to run two instances on the same port, and the shutdown hook for the second instance that does not actually bind to the port inadvertently breaks the first instance.

          Show
          Hoss Man added a comment - FYI: I think i may have found one possible cause of this problem .. not sure if there are other causes. 1) open two terminals both of which are in the solr example directory. 2) in terminal A, run 'java -jar start.jar' 3) note that the following URLs all work fine... http://localhost:8983/solr/ http://localhost:8983/solr/select/?q=solr&version=2.2&start=0&rows=10&indent=on http://localhost:8983/solr/admin/ 4) in terminal B, run "java -jar start.jar" 5) note that in terminal b, the server logs a "java.net.BindException: Address already in use" exception but the process does not terminate itself. 6) note that while the process in terminal B is still running, the urls from step#3 still work fine, and the queries are logged to terminal A 7) hit Ctrl-C in terminal B to kill that process, wait for it to log "Shutdown hook complete" and return your prompt 8) note that the following URLs now 404... http://localhost:8983/solr/admin/ http://localhost:8983/solr/ 9) note that this URL (and all non JSP non flat file solr urls) continues to work, and log the request to terminal A... http://localhost:8983/solr/select/?q=solr&version=2.2&start=0&rows=10&indent=on ..in short, i suspect this is a jetty problem, relating to people inadvertently trying to run two instances on the same port, and the shutdown hook for the second instance that does not actually bind to the port inadvertently breaks the first instance.
          Hide
          Bertrand Delacretaz added a comment -

          I'm seeing another, probably related, problem, on a Solr instance that's been running for 23 days: solr/admin/stats.xsl returns a 404 error, so although solr/admin/stats.jsp returns the correct XML it is not displayed in my browser.

          It seems like the default servlet (which serves static files) is dead, although the rest works fine. Most probably a Jetty issue.

          Show
          Bertrand Delacretaz added a comment - I'm seeing another, probably related, problem, on a Solr instance that's been running for 23 days: solr/admin/stats.xsl returns a 404 error, so although solr/admin/stats.jsp returns the correct XML it is not displayed in my browser. It seems like the default servlet (which serves static files) is dead, although the rest works fine. Most probably a Jetty issue.
          Hide
          Bertrand Delacretaz added a comment -

          > I suspect ....that it was a JSP issue with Jetty.

          Yes, certainly. Nothing seems to indicate a problem in Solr's code.

          Show
          Bertrand Delacretaz added a comment - > I suspect ....that it was a JSP issue with Jetty. Yes, certainly. Nothing seems to indicate a problem in Solr's code.
          Hide
          Hoss Man added a comment -

          FYI: there was more tothat orriginal thread then the apache archives show (because they are split up by month) here's the full discussion...

          http://www.nabble.com/Admin-page-went-down-tf2548760.html#a7103716

          ...at the time i wasn't able to reproduce the problem, but i wasn't hammering the port very hard. I suspect heavily that since hte problem was only with the admin pages, and all of the update/query functionality still worked fine that it was a JSP issue with Jetty.

          Show
          Hoss Man added a comment - FYI: there was more tothat orriginal thread then the apache archives show (because they are split up by month) here's the full discussion... http://www.nabble.com/Admin-page-went-down-tf2548760.html#a7103716 ...at the time i wasn't able to reproduce the problem, but i wasn't hammering the port very hard. I suspect heavily that since hte problem was only with the admin pages, and all of the update/query functionality still worked fine that it was a JSP issue with Jetty.
          Hide
          Bertrand Delacretaz added a comment -

          Upgrading is probably a good idea, at least to a released 5.x version, as apparently we're using a release candidate.

          Show
          Bertrand Delacretaz added a comment - Upgrading is probably a good idea, at least to a released 5.x version, as apparently we're using a release candidate.
          Hide
          Yonik Seeley added a comment -

          Maybe it's time to upgrade to the latest Jetty, or at least start evaluating it?
          That would also remove the requirement for a JDK over a JRE, and speed up JSP page compilation too.

          Show
          Yonik Seeley added a comment - Maybe it's time to upgrade to the latest Jetty, or at least start evaluating it? That would also remove the requirement for a JDK over a JRE, and speed up JSP page compilation too.
          Hide
          Bertrand Delacretaz added a comment -

          Yes, this is the Jetty that is bundled with Solr, Jetty/5.1.11RC0 according to the "Server" HTTP header.

          I haven't investigated on the Jetty side yet, it might be a known bug there

          Show
          Bertrand Delacretaz added a comment - Yes, this is the Jetty that is bundled with Solr, Jetty/5.1.11RC0 according to the "Server" HTTP header. I haven't investigated on the Jetty side yet, it might be a known bug there
          Hide
          Yonik Seeley added a comment -

          What version of jetty was it? The one included with Solr?

          I don't personally have experience with Solr + Jetty and long uptimes. We use Resin in-house, and don't have any uptime issues.

          Show
          Yonik Seeley added a comment - What version of jetty was it? The one included with Solr? I don't personally have experience with Solr + Jetty and long uptimes. We use Resin in-house, and don't have any uptime issues.
          Bertrand Delacretaz created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Bertrand Delacretaz
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development