HBase
  1. HBase
  2. HBASE-3900

Expose progress of a major compaction in UI and/or in shell

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.0
    • Component/s: None
    • Labels:
    • Release Note:
      Expose counts of KVs compacted to UI.
    • Tags:
      compaction, ui

      Description

      A general recommendation is to turn off major compactions and run them externally only currently the only way to follow progress of a major compaction is by study of regionserver logs. Add a feature that gives a percentage complete of major compaction up in the UI or in shell.

      1. 3900.addendum2
        1 kB
        Ted Yu
      2. HBASE-3900.patch
        22 kB
        Brad Anderson

        Activity

        Hide
        Brad Anderson added a comment -

        I've done a bit of work on this here: https://github.com/boorad/hbase/commit/51cdcb14680275a74a949a79b5f9b27c844be11c It's my first patch for hbase, so feedback is welcome.

        Talking with stack in irc, and he expressed a desire to see the compaction queue (ordered) with an identifier & whether or not it is a major compaction. I left this out of the above patch, as I am planning on reworking the existing web ui to be a bit friendlier to devs and ops folks.

        Show
        Brad Anderson added a comment - I've done a bit of work on this here: https://github.com/boorad/hbase/commit/51cdcb14680275a74a949a79b5f9b27c844be11c It's my first patch for hbase, so feedback is welcome. Talking with stack in irc, and he expressed a desire to see the compaction queue (ordered) with an identifier & whether or not it is a major compaction. I left this out of the above patch, as I am planning on reworking the existing web ui to be a bit friendlier to devs and ops folks.
        Hide
        stack added a comment -

        I added comments over on github repeated here:

        @brad Patch looks great. Missing license on CompactionProgress (I can add this on commit). Will we overflow the ints used keeping counts of KVs? Should this class and its methods be public since its only used in this package? Good stuff.

        Show
        stack added a comment - I added comments over on github repeated here: @brad Patch looks great. Missing license on CompactionProgress (I can add this on commit). Will we overflow the ints used keeping counts of KVs? Should this class and its methods be public since its only used in this package? Good stuff.
        Hide
        Brad Anderson added a comment -

        Updates from conversation at github:

        1. Missing license on CompactionProgress was done in subsequent commits

        2. Overflow of ints was addressed, changed to longs, tests pass

        3. To be visible to regionserver package, the regionserver.compactions package class I've added needs to be public

        4. Do the counters ever get reset? - yes, because Store.compactStore/4 gets a new CompactionProgress instance.

        Patch is attached, and last svn commit was trunk@1145459

        Show
        Brad Anderson added a comment - Updates from conversation at github: 1. Missing license on CompactionProgress was done in subsequent commits 2. Overflow of ints was addressed, changed to longs, tests pass 3. To be visible to regionserver package, the regionserver.compactions package class I've added needs to be public 4. Do the counters ever get reset? - yes, because Store.compactStore/4 gets a new CompactionProgress instance. Patch is attached, and last svn commit was trunk@1145459
        Hide
        Brad Anderson added a comment -

        here's the proper file.

        Show
        Brad Anderson added a comment - here's the proper file.
        Hide
        stack added a comment -

        Nice feature. Thank you Brad (Sorry it took a while to add – I added you as contributor and assigned you this issue). Committed to TRUNK.

        Show
        stack added a comment - Nice feature. Thank you Brad (Sorry it took a while to add – I added you as contributor and assigned you this issue). Committed to TRUNK.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2151 (See https://builds.apache.org/job/HBase-TRUNK/2151/)
        HBASE-3900 Expose progress of a major compaction in UI and/or in shell
        HBASE-3900 Expose progress of a major compaction in UI and/or in shell

        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java

        stack :
        Files :

        • /hbase/trunk/CHANGES.txt
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HServerLoad.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionRequest.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2151 (See https://builds.apache.org/job/HBase-TRUNK/2151/ ) HBASE-3900 Expose progress of a major compaction in UI and/or in shell HBASE-3900 Expose progress of a major compaction in UI and/or in shell stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java stack : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HServerLoad.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionRequest.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2152 (See https://builds.apache.org/job/HBase-TRUNK/2152/)
        HBASE-3900 Addendum, correct FIXED_OVERHEAD

        tedyu :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2152 (See https://builds.apache.org/job/HBase-TRUNK/2152/ ) HBASE-3900 Addendum, correct FIXED_OVERHEAD tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
        Hide
        Ted Yu added a comment -

        In build 2152, the following test failure is related to this JIRA (https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testMasterFailoverWithMockedRITOnDeadRS/):

        java.io.IOException: Error in readFields
        	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:584)
        	at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:64)
        	at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1152)
        	at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1112)
        	at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:669)
        	at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:463)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        	at java.lang.Thread.run(Thread.java:619)
        Caused by: A record version mismatch occured. Expecting v1, found v0
        	at org.apache.hadoop.io.VersionedWritable.readFields(VersionedWritable.java:46)
        	at org.apache.hadoop.hbase.HServerLoad$RegionLoad.readFields(HServerLoad.java:316)
        	at org.apache.hadoop.hbase.HServerLoad.readFields(HServerLoad.java:606)
        	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:581)
        	... 8 more
        
        Show
        Ted Yu added a comment - In build 2152, the following test failure is related to this JIRA ( https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testMasterFailoverWithMockedRITOnDeadRS/): java.io.IOException: Error in readFields at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:584) at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:64) at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1152) at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1112) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:669) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:463) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:619) Caused by: A record version mismatch occured. Expecting v1, found v0 at org.apache.hadoop.io.VersionedWritable.readFields(VersionedWritable.java:46) at org.apache.hadoop.hbase.HServerLoad$RegionLoad.readFields(HServerLoad.java:316) at org.apache.hadoop.hbase.HServerLoad.readFields(HServerLoad.java:606) at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:581) ... 8 more
        Hide
        Ted Yu added a comment -

        Here is addendum I committed:

        Index: src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
        ===================================================================
        --- src/main/java/org/apache/hadoop/hbase/regionserver/Store.java       (revision 1162317)
        +++ src/main/java/org/apache/hadoop/hbase/regionserver/Store.java       (working copy)
        @@ -1765,7 +1765,7 @@
           }
         
           public static final long FIXED_OVERHEAD = ClassSize.align(
        -      ClassSize.OBJECT + (15 * ClassSize.REFERENCE) +
        +      ClassSize.OBJECT + (16 * ClassSize.REFERENCE) +
               (8 * Bytes.SIZEOF_LONG) + (1 * Bytes.SIZEOF_DOUBLE) +
               (6 * Bytes.SIZEOF_INT) + (3 * Bytes.SIZEOF_BOOLEAN));
        
        Show
        Ted Yu added a comment - Here is addendum I committed: Index: src/main/java/org/apache/hadoop/hbase/regionserver/Store.java =================================================================== --- src/main/java/org/apache/hadoop/hbase/regionserver/Store.java (revision 1162317) +++ src/main/java/org/apache/hadoop/hbase/regionserver/Store.java (working copy) @@ -1765,7 +1765,7 @@ } public static final long FIXED_OVERHEAD = ClassSize.align( - ClassSize.OBJECT + (15 * ClassSize.REFERENCE) + + ClassSize.OBJECT + (16 * ClassSize.REFERENCE) + (8 * Bytes.SIZEOF_LONG) + (1 * Bytes.SIZEOF_DOUBLE) + (6 * Bytes.SIZEOF_INT) + (3 * Bytes.SIZEOF_BOOLEAN));
        Hide
        Ted Yu added a comment -

        2. Overflow of ints was addressed, changed to longs, tests pass

        Maybe due to the above change, I see inconsistent handling of the new fields. In readFields():

              this.totalCompactingKVs = in.readInt();
              this.currentCompactedKVs = in.readInt();
        

        In write(),

              out.writeLong(totalCompactingKVs);
              out.writeLong(currentCompactedKVs);
        
        Show
        Ted Yu added a comment - 2. Overflow of ints was addressed, changed to longs, tests pass Maybe due to the above change, I see inconsistent handling of the new fields. In readFields(): this .totalCompactingKVs = in.readInt(); this .currentCompactedKVs = in.readInt(); In write(), out.writeLong(totalCompactingKVs); out.writeLong(currentCompactedKVs);
        Hide
        Ted Yu added a comment -

        Here is attempt at fixing readFields

        Show
        Ted Yu added a comment - Here is attempt at fixing readFields
        Hide
        stack added a comment -

        +1 on patch Ted.

        Show
        stack added a comment - +1 on patch Ted.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2154 (See https://builds.apache.org/job/HBase-TRUNK/2154/)
        HBASE-3900 Addendum 2: try to fix error in readFields()

        tedyu :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HServerLoad.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2154 (See https://builds.apache.org/job/HBase-TRUNK/2154/ ) HBASE-3900 Addendum 2: try to fix error in readFields() tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HServerLoad.java
        Hide
        Ted Yu added a comment -

        In build 2154, I didn't see exception related to HServerLoad deserialization.
        The TestMergeTool.testMergeTool failure was due to the following check:

              HBaseAdmin.checkHBaseAvailable(getConf());
              LOG.fatal("HBase cluster must be off-line.");
        

        HBase cluster from some other test(s) was hanging.

        Show
        Ted Yu added a comment - In build 2154, I didn't see exception related to HServerLoad deserialization. The TestMergeTool.testMergeTool failure was due to the following check: HBaseAdmin.checkHBaseAvailable(getConf()); LOG.fatal( "HBase cluster must be off-line." ); HBase cluster from some other test(s) was hanging.

          People

          • Assignee:
            Brad Anderson
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development