Solr
  1. Solr
  2. SOLR-3972

Missing admin-extra files result in SEVERE log entries with giant stacktrace

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0, 4.1
    • Fix Version/s: 4.1, 6.0
    • Component/s: web gui
    • Labels:
      None
    • Environment:

      Description

      Missing admin-extra files result in SEVERE log entries with giant stacktrace.

      If a log entry is warranted at all, it should just be a one-line warning.

      1. SOLR-3972.patch
        5 kB
        Hoss Man
      2. SOLR-3972.patch
        4 kB
        Upayavira

        Issue Links

          Activity

          Hide
          Shawn Heisey added a comment -

          Full log/stacktrace:

          Oct 19, 2012 6:38:15 PM org.apache.solr.common.SolrException log
          SEVERE: org.apache.solr.common.SolrException: Can not find: admin-extra.menu-bottom.html [/index/solr4/cores/inc_0/conf/admin-extra.menu-bottom.html]
          at org.apache.solr.handler.admin.ShowFileRequestHandler.showFromFileSystem(ShowFileRequestHandler.java:231)
          at org.apache.solr.handler.admin.ShowFileRequestHandler.handleRequestBody(ShowFileRequestHandler.java:122)
          at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
          at org.apache.solr.core.SolrCore.execute(SolrCore.java:1750)
          at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
          at org.eclipse.jetty.server.Server.handle(Server.java:351)
          at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
          at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
          at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890)
          at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944)
          at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
          at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
          at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
          at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
          at java.lang.Thread.run(Thread.java:722)

          Show
          Shawn Heisey added a comment - Full log/stacktrace: Oct 19, 2012 6:38:15 PM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Can not find: admin-extra.menu-bottom.html [/index/solr4/cores/inc_0/conf/admin-extra.menu-bottom.html] at org.apache.solr.handler.admin.ShowFileRequestHandler.showFromFileSystem(ShowFileRequestHandler.java:231) at org.apache.solr.handler.admin.ShowFileRequestHandler.handleRequestBody(ShowFileRequestHandler.java:122) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1750) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:351) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:722)
          Hide
          Shawn Heisey added a comment -

          There is a workaround. Just create 0 byte files in the conf directory with these names:

          admin-extra.html
          admin-extra.menu-bottom.html
          admin-extra.menu-top.html

          The unix touch command works great for this.

          Show
          Shawn Heisey added a comment - There is a workaround. Just create 0 byte files in the conf directory with these names: admin-extra.html admin-extra.menu-bottom.html admin-extra.menu-top.html The unix touch command works great for this.
          Hide
          Shawn Heisey added a comment -

          As someone on the mailing list pointed out, my workaround is a poor one. It does work, and gets rid of the glaring error message, but it complicates HOWTO sort of documentation and is difficult for a novice to grasp.

          Show
          Shawn Heisey added a comment - As someone on the mailing list pointed out, my workaround is a poor one. It does work, and gets rid of the glaring error message, but it complicates HOWTO sort of documentation and is difficult for a novice to grasp.
          Hide
          Eric Pugh added a comment -

          Any reason to not just put in some blank files for the example app? With maybe a HTML comment telling folks to put in their own custom links etc? Why leave it to be magic? And when folks are browsing the /conf directory they will see these files and immediately grasp what they are for! That would be much simpler then having complex logic in the ShowFileHandler that decides when a missing a file is a INFO level error and when it is a WARNing.

          Show
          Eric Pugh added a comment - Any reason to not just put in some blank files for the example app? With maybe a HTML comment telling folks to put in their own custom links etc? Why leave it to be magic? And when folks are browsing the /conf directory they will see these files and immediately grasp what they are for! That would be much simpler then having complex logic in the ShowFileHandler that decides when a missing a file is a INFO level error and when it is a WARNing.
          Hide
          Alexandre Rafalovitch added a comment -

          This would work for an example up, but suddenly increases the minimum number of files for a solr instance from 3 (solr.xml, solrconfig.xml, schema.xml) to 5 (6?).

          Actually, I am not sure where the files themselves are requested from. Is it from UI/Ajax? Or somewhere from a server logic?

          Show
          Alexandre Rafalovitch added a comment - This would work for an example up, but suddenly increases the minimum number of files for a solr instance from 3 (solr.xml, solrconfig.xml, schema.xml) to 5 (6?). Actually, I am not sure where the files themselves are requested from. Is it from UI/Ajax? Or somewhere from a server logic?
          Hide
          Eric Pugh added a comment -

          It's from the new UI, it makes Ajax calls to load these files. And I guess I don't think of them as "required", since if your core doesn't have them, then things would still work, you would just get the error. I do think that having them in the ./example app would make it easier for folks to actually adopt that functionality.

          Show
          Eric Pugh added a comment - It's from the new UI, it makes Ajax calls to load these files. And I guess I don't think of them as "required", since if your core doesn't have them, then things would still work, you would just get the error. I do think that having them in the ./example app would make it easier for folks to actually adopt that functionality.
          Hide
          Alexandre Rafalovitch added a comment -

          The problem is that we don't have just 'example' instance, the problem will show up in DIH example for each core. So, it is a lot of empty files lying around. And, like I mentioned elsewhere, they are not even very useful, because all the formatting seems to be reset by the CSS.

          If we really do want them, can we request them with optional flag (e.g. ?optional=true) and check for that in file handler?

          Show
          Alexandre Rafalovitch added a comment - The problem is that we don't have just 'example' instance, the problem will show up in DIH example for each core. So, it is a lot of empty files lying around. And, like I mentioned elsewhere, they are not even very useful, because all the formatting seems to be reset by the CSS. If we really do want them, can we request them with optional flag (e.g. ?optional=true) and check for that in file handler?
          Hide
          Shawn Heisey added a comment -

          I agree with Alexandre. When I was copying from the example, which included copying the included jetty and creating an init script for it, I looked at the three extra html files, which ARE included there.

          I figured since they were "-extra" they were not necessary. I did look into their actual content and found it useless, so I didn't copy them to my installation. That's when I ran into the stacktrace - the 20 cores I had when I first set it up (deriving from my 3.x config) produced sixty of these huge log entries, even when logging at WARN. Now I have 16 cores on the server and my workaround has eliminated the error.

          If a fix for this issue were implemented (using log.warn()) and I deleted the empty html files, I would end up with 48 consecutive single log line entries if I left logging at WARN. If the logging were at INFO (which I believe is the default), they would probably be scattered throughout the log and not make much difference in its size. That's actually a good argument for making the message log at info and not warn.

          Show
          Shawn Heisey added a comment - I agree with Alexandre. When I was copying from the example, which included copying the included jetty and creating an init script for it, I looked at the three extra html files, which ARE included there. I figured since they were "-extra" they were not necessary. I did look into their actual content and found it useless, so I didn't copy them to my installation. That's when I ran into the stacktrace - the 20 cores I had when I first set it up (deriving from my 3.x config) produced sixty of these huge log entries, even when logging at WARN. Now I have 16 cores on the server and my workaround has eliminated the error. If a fix for this issue were implemented (using log.warn()) and I deleted the empty html files, I would end up with 48 consecutive single log line entries if I left logging at WARN. If the logging were at INFO (which I believe is the default), they would probably be scattered throughout the log and not make much difference in its size. That's actually a good argument for making the message log at info and not warn.
          Hide
          Shawn Heisey added a comment -

          It looks like the only non-test place that these filenames are mentioned is in a javascript file, and the error comes from core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java. It shows up twice - once when pulling from Zookeeper, and once when pulling from the filesystem.

          I will take a deeper look and attempt a patch when I make it to work and get it pulled into eclipse.

          Show
          Shawn Heisey added a comment - It looks like the only non-test place that these filenames are mentioned is in a javascript file, and the error comes from core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java. It shows up twice - once when pulling from Zookeeper, and once when pulling from the filesystem. I will take a deeper look and attempt a patch when I make it to work and get it pulled into eclipse.
          Hide
          Erick Erickson added a comment -

          Let's just nuke them. Frankly, this seems like one of those features that someone can implement in their very own installation if it's important. Solr has never been about providing a user-facing interface, the admin UI is a dev tool.

          I'd (grudgingly) go along with hardening the code to never log an error if these files aren't found, but inserting some junk like "if (filename == admin-extras.html) don't log" is really ugly in the code. All to make up for something that, IMO, shouldn't be there in the first place.

          FWIW
          Erick

          Show
          Erick Erickson added a comment - Let's just nuke them. Frankly, this seems like one of those features that someone can implement in their very own installation if it's important. Solr has never been about providing a user-facing interface, the admin UI is a dev tool. I'd (grudgingly) go along with hardening the code to never log an error if these files aren't found, but inserting some junk like "if (filename == admin-extras.html) don't log" is really ugly in the code. All to make up for something that, IMO, shouldn't be there in the first place. FWIW Erick
          Hide
          Hoss Man added a comment -

          I'm pretty sure this was already fixed as part of SOLR-4019 ... if not, then builidng off of those changes the fix should be easy. i'll do some testing

          Show
          Hoss Man added a comment - I'm pretty sure this was already fixed as part of SOLR-4019 ... if not, then builidng off of those changes the fix should be easy. i'll do some testing
          Hide
          Upayavira added a comment -

          This has to be one of the easier patches. This removes the admin-extra functionality from the admin UI.

          This patch appears to apply to both 4x and trunk.

          Show
          Upayavira added a comment - This has to be one of the easier patches. This removes the admin-extra functionality from the admin UI. This patch appears to apply to both 4x and trunk.
          Hide
          Hoss Man added a comment -

          This removes the admin-extra functionality from the admin UI.

          Upayavira: as mentioned in my previous comment, the idea behind the changes in SOLR-4019 was to eliminate these exceptions from the log, that patch just didn't address all of the code paths in ShowFileRequestHandler. While your patch completely removes the admin-extra UI feature, and eliminates the exceptions when using the UI – it doesn't address the underlying issue: ShowFileRequestHandler shouldn't log a big loud ugly stack trace for the understandable error of a 404 "file not found"

          This new patch fixes the remaining code paths in ShowFileREquestHandler to "return" the 404 exception instead of throwing it and includes tests for this – which should be done regardless of anything else. It also leaves the admin-extra functionality in place, since it's no longer noisy if the files 404.

          Does any one actually object to leaving this admin-extras functionality in place with this fix?

          Show
          Hoss Man added a comment - This removes the admin-extra functionality from the admin UI. Upayavira: as mentioned in my previous comment, the idea behind the changes in SOLR-4019 was to eliminate these exceptions from the log, that patch just didn't address all of the code paths in ShowFileRequestHandler. While your patch completely removes the admin-extra UI feature, and eliminates the exceptions when using the UI – it doesn't address the underlying issue: ShowFileRequestHandler shouldn't log a big loud ugly stack trace for the understandable error of a 404 "file not found" This new patch fixes the remaining code paths in ShowFileREquestHandler to "return" the 404 exception instead of throwing it and includes tests for this – which should be done regardless of anything else. It also leaves the admin-extra functionality in place, since it's no longer noisy if the files 404. Does any one actually object to leaving this admin-extras functionality in place with this fix?
          Hide
          Upayavira added a comment -

          I don't object to leaving the code in place. Our JIRA comments crossed and I didn't see yours until straight after I clicked 'go'.

          Given the exceptions have been fixed, the principal incentive to remove this code has been taken away. Good luck to anyone who finds out the functionality exists

          Show
          Upayavira added a comment - I don't object to leaving the code in place. Our JIRA comments crossed and I didn't see yours until straight after I clicked 'go'. Given the exceptions have been fixed, the principal incentive to remove this code has been taken away. Good luck to anyone who finds out the functionality exists
          Hide
          Shawn Heisey added a comment -

          Does any one actually object to leaving this admin-extras functionality in place with this fix?

          I do like the anmin-extras functionality. Just because I don't have a use for it right now doesn't mean that I won't in the future - it would make it a lot easier to hand off some admin responsibilities to untrained personnel if I could put docs and links for common tasks right in the UI.

          I filed the issue to eliminate the incredible noise in the logs. Looking at the patch, I can't tell what happens in the log. I'll test it later when I have a free moment. IMHO it ought to be logged at INFO, but WARN would be understandable too. Thanks!

          Show
          Shawn Heisey added a comment - Does any one actually object to leaving this admin-extras functionality in place with this fix? I do like the anmin-extras functionality. Just because I don't have a use for it right now doesn't mean that I won't in the future - it would make it a lot easier to hand off some admin responsibilities to untrained personnel if I could put docs and links for common tasks right in the UI. I filed the issue to eliminate the incredible noise in the logs. Looking at the patch, I can't tell what happens in the log. I'll test it later when I have a free moment. IMHO it ought to be logged at INFO, but WARN would be understandable too. Thanks!
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Chris M. Hostetter
          http://svn.apache.org/viewvc?view=revision&revision=1425207

          SOLR-3972: Fix ShowFileRequestHandler to not log a warning in the (expected) situation of a file not found.

          Show
          Commit Tag Bot added a comment - [trunk commit] Chris M. Hostetter http://svn.apache.org/viewvc?view=revision&revision=1425207 SOLR-3972 : Fix ShowFileRequestHandler to not log a warning in the (expected) situation of a file not found.
          Hide
          Hoss Man added a comment -

          Thanks!

          No, thank you for creating the issue – i really thought this was already fixed by SOLR-4019 or i would have dealt with it already.

          Show
          Hoss Man added a comment - Thanks! No, thank you for creating the issue – i really thought this was already fixed by SOLR-4019 or i would have dealt with it already.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Chris M. Hostetter
          http://svn.apache.org/viewvc?view=revision&revision=1425213

          SOLR-3972: Fix ShowFileRequestHandler to not log a warning in the (expected) situation of a file not found. (merge r1425207)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Chris M. Hostetter http://svn.apache.org/viewvc?view=revision&revision=1425213 SOLR-3972 : Fix ShowFileRequestHandler to not log a warning in the (expected) situation of a file not found. (merge r1425207)
          Hide
          Alexandre Rafalovitch added a comment -

          I do like the admin extra files as well, if they don't cause the exception. But if we are keeping them, can somebody have a look at Admin CSS as well (as a separate issue probably) and see why all the formatting in those admin extra files seem to be reset (e.g. H1, H2 all look the same as body text).

          Show
          Alexandre Rafalovitch added a comment - I do like the admin extra files as well, if they don't cause the exception. But if we are keeping them, can somebody have a look at Admin CSS as well (as a separate issue probably) and see why all the formatting in those admin extra files seem to be reset (e.g. H1, H2 all look the same as body text).

            People

            • Assignee:
              Hoss Man
              Reporter:
              Shawn Heisey
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development