HBase
  1. HBase
  2. HBASE-10213

Add read log size per second metrics for replication source

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.94.14
    • Fix Version/s: 0.98.0, 0.99.0
    • Component/s: metrics, Replication
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The current metrics of replication source contain logEditsReadRate, shippedBatchesRate, etc, which could indicate how fast the data replicated to peer cluster to some extent. However, it is not clear enough to know how many bytes replicating to peer cluster from these metrics. In production environment, it may be important to know the size of replicating data per second because the services may be affected if the network become busy.

      1. HBASE-10213-trunk-v1.patch
        3 kB
        cuijianwei
      2. HBASE-10213-0.94-v2.patch
        2 kB
        cuijianwei
      3. HBASE-10213-0.94-v1.patch
        2 kB
        cuijianwei
      4. 10213-trunk-addendum-1.patch
        1 kB
        Andrew Purtell

        Activity

        Hide
        cuijianwei added a comment -

        This patch adds a metric 'logReadRateInByte' to show how many bytes read by the source per second.

        Show
        cuijianwei added a comment - This patch adds a metric 'logReadRateInByte' to show how many bytes read by the source per second.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12619783/HBASE-10213-0.94-v1.patch
        against trunk revision .
        ATTACHMENT ID: 12619783

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8242//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619783/HBASE-10213-0.94-v1.patch against trunk revision . ATTACHMENT ID: 12619783 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8242//console This message is automatically generated.
        Hide
        Andrew Purtell added a comment -

        To get a good HadoopQA result, it will need a patch against trunk.

        index 3831bba..8315c3a 100644
        --- src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        +++ src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        @@ -458,6 +458,7 @@ public class ReplicationSource extends Thread
               throws IOException{
             long seenEntries = 0;
             this.repLogReader.seek();
        +    long persitionBeforeRead = this.repLogReader.getPosition();
             HLog.Entry entry =
                 this.repLogReader.readNextAndSetPosition();
             while (entry != null) {
        

        "persitionBeforeRead" should be "positionBeforeRead".

        index da0905c..e32a3bc 100644
        --- src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceMetrics.java
        +++ src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceMetrics.java
        @@ -66,6 +66,9 @@ public class ReplicationSourceMetrics implements Updater {
            */
           public final MetricsIntValue sizeOfLogQueue =
               new MetricsIntValue("sizeOfLogQueue", registry);
        +  
        +  /** Rate of log entries read by the source */
        +  public MetricsRate logReadRateInByte = new MetricsRate("logReadRateInByte", registry);
        

        The usual convention for names with units is to pluralize the unit, so "logReadRateInBytes"

        Show
        Andrew Purtell added a comment - To get a good HadoopQA result, it will need a patch against trunk. index 3831bba..8315c3a 100644 --- src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java +++ src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java @@ -458,6 +458,7 @@ public class ReplicationSource extends Thread throws IOException{ long seenEntries = 0; this .repLogReader.seek(); + long persitionBeforeRead = this .repLogReader.getPosition(); HLog.Entry entry = this .repLogReader.readNextAndSetPosition(); while (entry != null ) { "persitionBeforeRead" should be "positionBeforeRead". index da0905c..e32a3bc 100644 --- src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceMetrics.java +++ src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceMetrics.java @@ -66,6 +66,9 @@ public class ReplicationSourceMetrics implements Updater { */ public final MetricsIntValue sizeOfLogQueue = new MetricsIntValue( "sizeOfLogQueue" , registry); + + /** Rate of log entries read by the source */ + public MetricsRate logReadRateInByte = new MetricsRate( "logReadRateInByte" , registry); The usual convention for names with units is to pluralize the unit, so "logReadRateInBytes"
        Hide
        cuijianwei added a comment -

        Thanks for your comment Andrew Purtell. I update the patch for 0.94 and add a patch for trunk.

        Show
        cuijianwei added a comment - Thanks for your comment Andrew Purtell . I update the patch for 0.94 and add a patch for trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12620302/HBASE-10213-trunk-v1.patch
        against trunk revision .
        ATTACHMENT ID: 12620302

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12620302/HBASE-10213-trunk-v1.patch against trunk revision . ATTACHMENT ID: 12620302 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8277//console This message is automatically generated.
        Hide
        Andrew Purtell added a comment -
        Show
        Andrew Purtell added a comment - lgtm, Elliott Clark ?
        Hide
        Elliott Clark added a comment -

        +1 I need to go and make these not use the HashMap but this follows what's there now.

        Show
        Elliott Clark added a comment - +1 I need to go and make these not use the HashMap but this follows what's there now.
        Hide
        Ted Yu added a comment -

        Integrated to trunk.

        Thanks for the patch, Jianwei.

        Andrew Purtell: Do you want this in 0.98 ?

        Show
        Ted Yu added a comment - Integrated to trunk. Thanks for the patch, Jianwei. Andrew Purtell : Do you want this in 0.98 ?
        Hide
        Andrew Purtell added a comment -

        Committed to 0.98. Thanks Ted.

        I fixed a spelling error on commit. Attached is an addendum for trunk to match what went into 0.98. I committed this trivial change to trunk using CTR as r1554361.

        stack and Lars Hofhansl: We can close this after a decision on 0.96 and 0.94. Almost feel bad pinging you, should be no big deal.

        Show
        Andrew Purtell added a comment - Committed to 0.98. Thanks Ted. I fixed a spelling error on commit. Attached is an addendum for trunk to match what went into 0.98. I committed this trivial change to trunk using CTR as r1554361. stack and Lars Hofhansl : We can close this after a decision on 0.96 and 0.94. Almost feel bad pinging you, should be no big deal.
        Hide
        Ted Yu added a comment -

        Thanks for the catch, Andy.

        It would be not so good if a public method has spelling mistake.

        Show
        Ted Yu added a comment - Thanks for the catch, Andy. It would be not so good if a public method has spelling mistake.
        Hide
        cuijianwei added a comment -

        Thanks for the comment Andrew Purtell and Ted Yu. I should pay more attention on spelling.

        Show
        cuijianwei added a comment - Thanks for the comment Andrew Purtell and Ted Yu . I should pay more attention on spelling.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4770 (See https://builds.apache.org/job/HBase-TRUNK/4770/)
        HBASE-10213 Add read log size per second metrics for replication source (tedyu: rev 1554347)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4770 (See https://builds.apache.org/job/HBase-TRUNK/4770/ ) HBASE-10213 Add read log size per second metrics for replication source (tedyu: rev 1554347) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98 #46 (See https://builds.apache.org/job/HBase-0.98/46/)
        HBASE-10213. Add read log size per second metrics for replication source (cuijianwei) (apurtell: rev 1554359)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98 #46 (See https://builds.apache.org/job/HBase-0.98/46/ ) HBASE-10213 . Add read log size per second metrics for replication source (cuijianwei) (apurtell: rev 1554359) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4771 (See https://builds.apache.org/job/HBase-TRUNK/4771/)
        Amend HBASE-10213. Spelling fix for method o.a.h.h.replication.regionserver.MetricsSource#incrLogReadInBytes (apurtell: rev 1554361)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4771 (See https://builds.apache.org/job/HBase-TRUNK/4771/ ) Amend HBASE-10213 . Spelling fix for method o.a.h.h.replication.regionserver.MetricsSource#incrLogReadInBytes (apurtell: rev 1554361) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #30 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/30/)
        Amend HBASE-10213. Spelling fix for method o.a.h.h.replication.regionserver.MetricsSource#incrLogReadInBytes (apurtell: rev 1554361)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          HBASE-10213 Add read log size per second metrics for replication source (tedyu: rev 1554347)
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #30 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/30/ ) Amend HBASE-10213 . Spelling fix for method o.a.h.h.replication.regionserver.MetricsSource#incrLogReadInBytes (apurtell: rev 1554361) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java HBASE-10213 Add read log size per second metrics for replication source (tedyu: rev 1554347) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #44 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/44/)
        HBASE-10213. Add read log size per second metrics for replication source (cuijianwei) (apurtell: rev 1554359)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #44 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/44/ ) HBASE-10213 . Add read log size per second metrics for replication source (cuijianwei) (apurtell: rev 1554359) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
        Hide
        Andrew Purtell added a comment -

        This has been committed to trunk and 0.98 and this issue is now just sitting here unresolved. Resolving. Reopen for commit to another branch or open a new JIRA for a backport at your option.

        Show
        Andrew Purtell added a comment - This has been committed to trunk and 0.98 and this issue is now just sitting here unresolved. Resolving. Reopen for commit to another branch or open a new JIRA for a backport at your option.
        Hide
        Honghua Feng added a comment -

        However, it is not clear enough to know how many bytes replicating to peer cluster from these metrics. In production environment, it may be important to know the size of replicating data per second

        the intention of this jira is good, but by examining the patch:

        metrics.incrLogReadInByes(this.repLogReader.getPosition() - positionBeforeRead);

        the metric above only reflects the log read/parse rate, not the desired replicating data to peer cluster rate, since the read/parsed log files may contain many kvs from column-families with replication scope=0 which will be filtered out and removed from the entries list before the real replicating to peer cluster occurs...
        why not use currentSize, the size of all entries which will be really replicated to the peer cluster?

        Show
        Honghua Feng added a comment - However, it is not clear enough to know how many bytes replicating to peer cluster from these metrics. In production environment, it may be important to know the size of replicating data per second the intention of this jira is good , but by examining the patch: metrics.incrLogReadInByes( this .repLogReader.getPosition() - positionBeforeRead); the metric above only reflects the log read/parse rate, not the desired replicating data to peer cluster rate, since the read/parsed log files may contain many kvs from column-families with replication scope=0 which will be filtered out and removed from the entries list before the real replicating to peer cluster occurs... why not use currentSize, the size of all entries which will be really replicated to the peer cluster?
        Hide
        cuijianwei added a comment -

        Honghua Feng, thanks for your comment. We need to consider the filtered kvs when computing how much data replicate to peer cluster. The 'logReadRateInByte' only computes how much we read from HLog in source cluster. Maybe, we only another metrics such as 'logReplicateRateInByte'?

        Show
        cuijianwei added a comment - Honghua Feng , thanks for your comment. We need to consider the filtered kvs when computing how much data replicate to peer cluster. The 'logReadRateInByte' only computes how much we read from HLog in source cluster. Maybe, we only another metrics such as 'logReplicateRateInByte'?

          People

          • Assignee:
            cuijianwei
            Reporter:
            cuijianwei
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development