Solr
  1. Solr
  2. SOLR-2698

Enhance CoreAdmin STATUS command to return index size

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0-ALPHA
    • Component/s: multicore
    • Labels:
      None

      Description

      CoreAdmin STATUS command returns all kinds of index info for all cores on the server, except for the index size.
      However, indexSize can be retrieved for an individual core via a /replication&command=details request.

      I have N Solrs servers, running M cores each. My application is monitoring the status of all cores, including their index size.
      As it stands today, I need to issue N status requests plus N*M replication requests to get all the information I need.
      If STATUS command returned indexSize, number of requests would be just N.

      1. SOLR-2698.patch
        2 kB
        Yury Kats
      2. SOLR-2698.patch
        10 kB
        Mark Miller

        Activity

        Hide
        Mark Miller added a comment -

        Too late for backport - resolving.

        Show
        Mark Miller added a comment - Too late for backport - resolving.
        Hide
        Yury Kats added a comment -

        To clarify, this is already fixed in 4.0. The bug was re-opened for a potential backport to 3.x.

        Show
        Yury Kats added a comment - To clarify, this is already fixed in 4.0. The bug was re-opened for a potential backport to 3.x.
        Hide
        Jason Rutherglen added a comment -

        +1 This'd be useful.

        Show
        Jason Rutherglen added a comment - +1 This'd be useful.
        Hide
        Robert Muir added a comment -

        3.4 -> 3.5

        Show
        Robert Muir added a comment - 3.4 -> 3.5
        Hide
        Mark Miller added a comment -

        hmmm...reopening - might as well back port this to 3x

        Show
        Mark Miller added a comment - hmmm...reopening - might as well back port this to 3x
        Hide
        Mark Miller added a comment -

        Thanks Yury!

        Show
        Mark Miller added a comment - Thanks Yury!
        Hide
        Mark Miller added a comment -

        Here is a patch - I'll commit soon.

        Sorry for the delay. Hard to work while my eyes are bleeding from staring at my brokerage account

        Show
        Mark Miller added a comment - Here is a patch - I'll commit soon. Sorry for the delay. Hard to work while my eyes are bleeding from staring at my brokerage account
        Hide
        Yury Kats added a comment -

        Mark, is this good to go or are you waiting for an updated patch from me? Just double checking.

        Show
        Yury Kats added a comment - Mark, is this good to go or are you waiting for an updated patch from me? Just double checking.
        Hide
        Yury Kats added a comment -

        I see the following JIRA issue as well: https://issues.apache.org/jira/browse/IO-226

        Funny how it was closed "not a bug". Who needs a method that returns "1 GB" for a 1.99GB file?

        humanReadableUnits looks good. Maybe add TB? Storage is getting cheaper by the day.

        I think both is nice myself - but of course not really necessary - a nicety though, in the case an application wants to display this, nice if we simply do the work for them.

        +1 on two values in the response. In CoreAdminHandler and ReplicationHandler.

        Show
        Yury Kats added a comment - I see the following JIRA issue as well: https://issues.apache.org/jira/browse/IO-226 Funny how it was closed "not a bug". Who needs a method that returns "1 GB" for a 1.99GB file? humanReadableUnits looks good. Maybe add TB? Storage is getting cheaper by the day. I think both is nice myself - but of course not really necessary - a nicety though, in the case an application wants to display this, nice if we simply do the work for them. +1 on two values in the response. In CoreAdminHandler and ReplicationHandler.
        Hide
        Mark Miller added a comment -

        We could go a step further and return index size in bytes AND as a readable string in the response.

        I think both is nice myself - but of course not really necessary - a nicety though, in the case an application wants to display this, nice if we simply do the work for them.

        If need to choose one or the other, my preference would be for just bytes (long).

        +1

        Show
        Mark Miller added a comment - We could go a step further and return index size in bytes AND as a readable string in the response. I think both is nice myself - but of course not really necessary - a nicety though, in the case an application wants to display this, nice if we simply do the work for them. If need to choose one or the other, my preference would be for just bytes (long). +1
        Hide
        Mark Miller added a comment -

        FileUtils.byteCountToDisplaySize( ) will not round the size of a file

        Ah, right, good catch! This is actually why I implemented our own version in one of our projects (similar to the one in rep handler). Had forgotten the reason for that! See below. I see the following JIRA issue as well: https://issues.apache.org/jira/browse/IO-226

        So yeah, +1 on the new Util method.

        
          /**
           * Return good default units based on byte size.
           */
          public static String humanReadableUnits(long bytes) {
            String newSizeAndUnits;
            DecimalFormat df = new DecimalFormat("#.#");
            if (bytes / ONE_GB > 0) {
              newSizeAndUnits = String.valueOf(df.format((float)bytes / ONE_GB)) + " GB";
            } else if (bytes / ONE_MB > 0) {
              newSizeAndUnits = String.valueOf(df.format((float)bytes / ONE_MB)) + " MB";
            } else if (bytes / ONE_KB > 0) {
              newSizeAndUnits = String.valueOf(df.format((float)bytes / ONE_KB)) + " KB";
            } else {
              newSizeAndUnits = String.valueOf(bytes) + " bytes";
            }
        
            return newSizeAndUnits;
          }
        
        Show
        Mark Miller added a comment - FileUtils.byteCountToDisplaySize( ) will not round the size of a file Ah, right, good catch! This is actually why I implemented our own version in one of our projects (similar to the one in rep handler). Had forgotten the reason for that! See below. I see the following JIRA issue as well: https://issues.apache.org/jira/browse/IO-226 So yeah, +1 on the new Util method. /** * Return good default units based on byte size. */ public static String humanReadableUnits( long bytes) { String newSizeAndUnits; DecimalFormat df = new DecimalFormat( "#.#" ); if (bytes / ONE_GB > 0) { newSizeAndUnits = String .valueOf(df.format(( float )bytes / ONE_GB)) + " GB" ; } else if (bytes / ONE_MB > 0) { newSizeAndUnits = String .valueOf(df.format(( float )bytes / ONE_MB)) + " MB" ; } else if (bytes / ONE_KB > 0) { newSizeAndUnits = String .valueOf(df.format(( float )bytes / ONE_KB)) + " KB" ; } else { newSizeAndUnits = String .valueOf(bytes) + " bytes" ; } return newSizeAndUnits; }
        Hide
        Yury Kats added a comment -

        FileUtils.byteCountToDisplaySize could produce a different result than what's being returned now.

        http://www.discursive.com/books/cjcook/reference/io-network-sect-printing-human-readable mentions:

        FileUtils.byteCountToDisplaySize( ) will not round the size of a file; a 2.9 MB file will have a display size of 2 MB.

        If that's true, it's not very useful.

        We could go a step further and return index size in bytes AND as a readable string in the response.
        If need to choose one or the other, my preference would be for just bytes (long). If it's a String, any application processing the response will need to convert it into bytes anyway to do any meaningful analysis.

        Show
        Yury Kats added a comment - FileUtils.byteCountToDisplaySize could produce a different result than what's being returned now. http://www.discursive.com/books/cjcook/reference/io-network-sect-printing-human-readable mentions: FileUtils.byteCountToDisplaySize( ) will not round the size of a file; a 2.9 MB file will have a display size of 2 MB. If that's true, it's not very useful. We could go a step further and return index size in bytes AND as a readable string in the response. If need to choose one or the other, my preference would be for just bytes (long). If it's a String, any application processing the response will need to convert it into bytes anyway to do any meaningful analysis.
        Hide
        Mark Miller added a comment -

        I'm very new to the solr codebase and thus just looked how the same function is already implemented.

        Yup, made perfect sense to see how it was done and copy it - just making an overall improvement suggestion.

        ReplicationHandler also converts size into a "readable" string.

        There is a commons method for this type of thing as well:

        org.apache.commons.io.FileUtils.byteCountToDisplaySize(size)

        Show
        Mark Miller added a comment - I'm very new to the solr codebase and thus just looked how the same function is already implemented. Yup, made perfect sense to see how it was done and copy it - just making an overall improvement suggestion. ReplicationHandler also converts size into a "readable" string. There is a commons method for this type of thing as well: org.apache.commons.io.FileUtils.byteCountToDisplaySize(size)
        Hide
        Yury Kats added a comment -

        I don't see why not. I'm very new to the solr codebase and thus just looked how the same function is already implemented.

        ReplicationHandler also converts size into a "readable" string. For consistency sake CoreAdminHandler could do the same (ReplicationHandler#readableSize could be a shared util).

        Show
        Yury Kats added a comment - I don't see why not. I'm very new to the solr codebase and thus just looked how the same function is already implemented. ReplicationHandler also converts size into a "readable" string. For consistency sake CoreAdminHandler could do the same (ReplicationHandler#readableSize could be a shared util).
        Hide
        Mark Miller added a comment -

        + // Two methods below are copied from ReplicationHandler.
        + // Could be refactored into an utility

        Shouldn't we simply use org.apache.commons.io.FileUtils.sizeOfDirectory(f) in both places ?

        Show
        Mark Miller added a comment - + // Two methods below are copied from ReplicationHandler. + // Could be refactored into an utility Shouldn't we simply use org.apache.commons.io.FileUtils.sizeOfDirectory(f) in both places ?
        Hide
        Yury Kats added a comment -

        This patch adds index size to CoreAdminHandler#getCoreStatus

        Show
        Yury Kats added a comment - This patch adds index size to CoreAdminHandler#getCoreStatus

          People

          • Assignee:
            Mark Miller
            Reporter:
            Yury Kats
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development