Solr
  1. Solr
  2. SOLR-4526

Admin UI depends on optional system info

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.1
    • Fix Version/s: 4.2, Trunk
    • Component/s: web gui
    • Labels:
      None
    • Environment:

      Oracle Corporation OpenJDK 64-Bit Server VM (1.7.0_15 23.7-b01)

      Description

      A user on IRC was having trouble getting file descriptor counts and JVM memory usage in the admin UI, but it worked perfectly fine on another system. The problem system uses OpenJDK, the other one uses the Apple JDK. The user had tracked it down to an exception while trying to get open file descriptor info.

      Looking in the SystemInfoHandler.java file, I see a comment reference to com.sun.management.UnixOperatingSystemMXBean at the point where it is getting file descriptor info. A little extra searching turned up ZOOKEEPER-1579 which refers to HBASE-6945, the same problem with OpenJDK.

      1. ubuntu-jetty6.png
        154 kB
        Felix Buenemann
      2. SOLR-4526.patch
        4 kB
        Felix Buenemann
      3. built-in-jetty8.png
        129 kB
        Felix Buenemann

        Issue Links

          Activity

          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Stefan Matheis
          http://svn.apache.org/viewvc?view=revision&revision=1452838

          SOLR-4526: Admin UI depends on optional system info (merge r1452835)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Stefan Matheis http://svn.apache.org/viewvc?view=revision&revision=1452838 SOLR-4526 : Admin UI depends on optional system info (merge r1452835)
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Stefan Matheis
          http://svn.apache.org/viewvc?view=revision&revision=1452835

          SOLR-4526: Admin UI depends on optional system info

          Show
          Commit Tag Bot added a comment - [trunk commit] Stefan Matheis http://svn.apache.org/viewvc?view=revision&revision=1452835 SOLR-4526 : Admin UI depends on optional system info
          Hide
          Felix Buenemann added a comment -

          Updated the patch, because the previous version contained a copy'n'paste error, causing jvm memory info to always be hidden.

          Show
          Felix Buenemann added a comment - Updated the patch, because the previous version contained a copy'n'paste error, causing jvm memory info to always be hidden.
          Hide
          Uwe Schindler added a comment -

          The patch should also apply cleanly against 4.1.0 with minor offset.

          That's not an issue, as we merge using SVN tools. Patches are only applied to trunk.

          Show
          Uwe Schindler added a comment - The patch should also apply cleanly against 4.1.0 with minor offset. That's not an issue, as we merge using SVN tools. Patches are only applied to trunk.
          Hide
          Felix Buenemann added a comment -

          Btw. if desired I can split out the changes to loadaverage matching into a different patch/ticket, but steffkes said on IRC it'd be ok to include.

          Show
          Felix Buenemann added a comment - Btw. if desired I can split out the changes to loadaverage matching into a different patch/ticket, but steffkes said on IRC it'd be ok to include.
          Hide
          Felix Buenemann added a comment - - edited

          The attached patch against trunk makes the dashboard js code less reliant on provided system and jvm data by checking for their existance. It also improves the matching of load average among operating systems (should now work with Linux, Mac OS X, *BSD, Irix, etc.)

          The patch should also apply cleanly against 4.1.0 with minor offset.

          Show
          Felix Buenemann added a comment - - edited The attached patch against trunk makes the dashboard js code less reliant on provided system and jvm data by checking for their existance. It also improves the matching of load average among operating systems (should now work with Linux, Mac OS X, *BSD, Irix, etc.) The patch should also apply cleanly against 4.1.0 with minor offset.
          Hide
          Uwe Schindler added a comment -

          You cannot rely on this data because it is very JVM specific, so we have to handle it proper. As said before, IBJ J9 and Oracle JRockit fail to display anything in that area (because of the bugs in index.js).

          Show
          Uwe Schindler added a comment - You cannot rely on this data because it is very JVM specific, so we have to handle it proper. As said before, IBJ J9 and Oracle JRockit fail to display anything in that area (because of the bugs in index.js).
          Hide
          Felix Buenemann added a comment -

          I could provide a patch against index.js to remove the hard dependencies, if there is consensus that this data should not be relied on. However it needs to be decided what to display in this case (eg. hide the bar, show "Data not available", ...).

          Show
          Felix Buenemann added a comment - I could provide a patch against index.js to remove the hard dependencies, if there is consensus that this data should not be relied on. However it needs to be decided what to display in this case (eg. hide the bar, show "Data not available", ...).
          Hide
          Uwe Schindler added a comment -

          The general problem here is the admin UI, as mentioned by Felix. The output of the MX data is very JVM-specific, so the UI should not rely on key/value pairs are always available. If you e.g. start Solr with IBM J9, the whole Andmin UI shows nothing at all for those statistics, same for JRockit. If you look into browser JS logs you see errors about missing keys.

          The RequestHandler on the Solr side is fine and produces no log entries. Because of that you will see no errors in the log file, as the UI cannot log anything to the Solr log.

          The reason why the Ubuntu-Jetty does not display those information might be because the Webapp context is restricted and does not provide the OS-MXBean. The Jetty included with Solr has no security settings at all and the webapp is running in the root context.

          Show
          Uwe Schindler added a comment - The general problem here is the admin UI, as mentioned by Felix. The output of the MX data is very JVM-specific, so the UI should not rely on key/value pairs are always available. If you e.g. start Solr with IBM J9, the whole Andmin UI shows nothing at all for those statistics, same for JRockit. If you look into browser JS logs you see errors about missing keys. The RequestHandler on the Solr side is fine and produces no log entries. Because of that you will see no errors in the log file, as the UI cannot log anything to the Solr log. The reason why the Ubuntu-Jetty does not display those information might be because the Webapp context is restricted and does not provide the OS-MXBean. The Jetty included with Solr has no security settings at all and the webapp is running in the root context.
          Hide
          Felix Buenemann added a comment -

          Hoss: Solr is not logging any errors. I've tried with logging for solr and children set to INFO.

          Show
          Felix Buenemann added a comment - Hoss: Solr is not logging any errors. I've tried with logging for solr and children set to INFO.
          Hide
          Felix Buenemann added a comment -

          Hoss: I'll try to get my VM back into the prober state and check it.

          Regarding what the JS could do better: It would be a good idea to check if the accessed array keys are undefined, to avoid an exception. This would make the JS code more verbose though.
          Had the code checked for undefined values, it would have shown the JVM usage just fine.

          Show
          Felix Buenemann added a comment - Hoss: I'll try to get my VM back into the prober state and check it. Regarding what the JS could do better: It would be a good idea to check if the accessed array keys are undefined, to avoid an exception. This would make the JS code more verbose though. Had the code checked for undefined values, it would have shown the JVM usage just fine.
          Hide
          Hoss Man added a comment -

          Felix: can you confirm no errors in the server logs, just javacsript was having problems from the null keys?

          asigning to steffkes in the hopes that he can take a look – but based on the screen shot i'm not really sure what if anything it could do better in a situation like this – if the values aren't available from the JVM, showing "blank" bar sharts seems as good an option as any ... i guess maybe labeling them with "?" could be a little better though.

          Show
          Hoss Man added a comment - Felix: can you confirm no errors in the server logs, just javacsript was having problems from the null keys? asigning to steffkes in the hopes that he can take a look – but based on the screen shot i'm not really sure what if anything it could do better in a situation like this – if the values aren't available from the JVM, showing "blank" bar sharts seems as good an option as any ... i guess maybe labeling them with "?" could be a little better though.
          Hide
          Felix Buenemann added a comment -

          Screenshots illustrating the problem. The JVM values are only empty because of an exception in index.js, which was looking for the missing keys 'maxFileDescriptorCount' and 'openFileDescriptorCount' in system_data['system'].

          Show
          Felix Buenemann added a comment - Screenshots illustrating the problem. The JVM values are only empty because of an exception in index.js, which was looking for the missing keys 'maxFileDescriptorCount' and 'openFileDescriptorCount' in system_data ['system'] .
          Hide
          Felix Buenemann added a comment -

          The problem turned out to be jetty6 vs. jetty8 and not OpenJDK. The dev system was running Solr's built-in jetty, while the production system was running the jetty version that was shipping with Ubuntu 12.04 Server. I noticed this after building a VM to simulate the problem with the closed Oracle JDK7 and encountered the same behavior.

          I'm attaching two screenshots to show the problem.

          Show
          Felix Buenemann added a comment - The problem turned out to be jetty6 vs. jetty8 and not OpenJDK. The dev system was running Solr's built-in jetty, while the production system was running the jetty version that was shipping with Ubuntu 12.04 Server. I noticed this after building a VM to simulate the problem with the closed Oracle JDK7 and encountered the same behavior. I'm attaching two screenshots to show the problem.
          Hide
          Shawn Heisey added a comment -

          Additional note from the user on IRC: I adjusted the live server config to run the solr built-in jetty and now all the stats are showing up, even on openjdk, so the issue seems to be with older jetty versions.

          Show
          Shawn Heisey added a comment - Additional note from the user on IRC: I adjusted the live server config to run the solr built-in jetty and now all the stats are showing up, even on openjdk, so the issue seems to be with older jetty versions.
          Hide
          Shawn Heisey added a comment -

          I think perhaps I was a little too aggressive in framing this issue. I should have scaled it back a little bit to focus on the UI issue, not the java classes. The title may need changing, and it may require filing another issue. I see two problems. With OpenJDK, the file descriptor info and the JVM memory info are not showing on the dashboard. Side note: I don't have OpenJDK running anywhere and don't have unallocated hardware I can fiddle with right now.

          The underlying issue is that the user says the file descriptor info is not found in /admin/system output running under OpenJDK, but it is there under Apple or Oracle JVMs. If the info is available at all from OpenJDK, it may require a code change to get it.

          The java issue is triggering a javascript issue. An educated guess is that the code that populates the file descriptor bar is failing and causing the JVM memory bar calculation (JVM memory info is available in /admin/system) to be skipped entirely. A simple fix for that issue would be to re-order the javascript code so that jvm memory is done before the file descriptor. A better fix would be to make sure that a nonexistent piece of information doesn't cause problems.

          I'm very weak in both javascript and java reflection, or I would have already tried to come up with patches.

          Show
          Shawn Heisey added a comment - I think perhaps I was a little too aggressive in framing this issue. I should have scaled it back a little bit to focus on the UI issue, not the java classes. The title may need changing, and it may require filing another issue. I see two problems. With OpenJDK, the file descriptor info and the JVM memory info are not showing on the dashboard. Side note: I don't have OpenJDK running anywhere and don't have unallocated hardware I can fiddle with right now. The underlying issue is that the user says the file descriptor info is not found in /admin/system output running under OpenJDK, but it is there under Apple or Oracle JVMs. If the info is available at all from OpenJDK, it may require a code change to get it. The java issue is triggering a javascript issue. An educated guess is that the code that populates the file descriptor bar is failing and causing the JVM memory bar calculation (JVM memory info is available in /admin/system) to be skipped entirely. A simple fix for that issue would be to re-order the javascript code so that jvm memory is done before the file descriptor. A better fix would be to make sure that a nonexistent piece of information doesn't cause problems. I'm very weak in both javascript and java reflection, or I would have already tried to come up with patches.
          Hide
          Hoss Man added a comment -

          I'm confused: SystemInfoHandler doesn't directly depend on UnixOperatingSystemMXBean, it uses reflection to ensure that it only calls methods on the OperatingSystemMXBean if they already exist – so even if it isn't a sun JVM, or isn't an UnixOperatingSystemMXBean instance, or is an older version of UnixOperatingSystemMXBean that doesn't have some of those methods, it still shouldn't be an error – at worst those properties just won't be included in the response.

          perhaps the problem is just that that admin UI Javascript assumes those properties will always be available? (but that wouldn't explain the comment about "tracked it down to an exception while trying to get open file descriptor info")

          can anyone who can actually reproduce this problem please post the specifics of the exception they see, or the behavior they see in the admin ui (ie: screenshot)

          Show
          Hoss Man added a comment - I'm confused: SystemInfoHandler doesn't directly depend on UnixOperatingSystemMXBean, it uses reflection to ensure that it only calls methods on the OperatingSystemMXBean if they already exist – so even if it isn't a sun JVM, or isn't an UnixOperatingSystemMXBean instance, or is an older version of UnixOperatingSystemMXBean that doesn't have some of those methods, it still shouldn't be an error – at worst those properties just won't be included in the response. perhaps the problem is just that that admin UI Javascript assumes those properties will always be available? (but that wouldn't explain the comment about "tracked it down to an exception while trying to get open file descriptor info") can anyone who can actually reproduce this problem please post the specifics of the exception they see, or the behavior they see in the admin ui (ie: screenshot)
          Hide
          Shawn Heisey added a comment -

          If File descriptor info just isn't available in OpenJDK, then that is life. The file descriptor problem seems to be causing another problem - the JVM memory info isn't showing up on the dashboard. I haven't looked at the code yet, and I don't know javascript very well, but the user suspects that when the file descriptor info is not found, it causes the code that puts the JVM memory info together to be skipped.

          Show
          Shawn Heisey added a comment - If File descriptor info just isn't available in OpenJDK, then that is life. The file descriptor problem seems to be causing another problem - the JVM memory info isn't showing up on the dashboard. I haven't looked at the code yet, and I don't know javascript very well, but the user suspects that when the file descriptor info is not found, it causes the code that puts the JVM memory info together to be skipped.

            People

            • Assignee:
              Stefan Matheis (steffkes)
              Reporter:
              Shawn Heisey
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development