[HBASE-11617] incorrect AgeOfLastAppliedOp and AgeOfLastShippedOp in replication Metrics when no new replication OP

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.98.2
• Fix Version/s: 0.99.0, 0.98.6
• Component/s: Replication
• Hadoop Flags: Reviewed
• Assignee: Demai Ni
• Reporter: Demai Ni
• Created: 30/Jul/14 18:11
• Updated: 06/Apr/18 17:51
• Resolved: 23/Aug/14 16:47

Description

AgeOfLastAppliedOp in MetricsSink.java is to indicate the time an edit sat in the 'replication queue' before it got replicated(aka applied)

  /**
   * Set the age of the last applied operation
   *
   * @param timestamp The timestamp of the last operation applied.
   * @return the age that was set
   */
  public long setAgeOfLastAppliedOp(long timestamp) {
    lastTimestampForAge = timestamp;
    long age = System.currentTimeMillis() - lastTimestampForAge;
    rms.setGauge(SINK_AGE_OF_LAST_APPLIED_OP, age);
    return age;
  } 

In the following scenario:
1) at 7:00am a sink op is applied, and the SINK_AGE_OF_LAST_APPLIED_OP is
set for example 100ms;
2) and then NO new Sink op occur.
3) when a refreshAgeOfLastAppliedOp() is invoked at 8:00am. Instead of
return the 100ms, the AgeOfLastAppliedOp become 1hour + 100ms,

It was because that refreshAgeOfLastAppliedOp() get invoked periodically by getStats().

proposed fix:

--- hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java
+++ hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java
@@ -35,6 +35,7 @@ public class MetricsSink {
 
   private MetricsReplicationSource rms;
   private long lastTimestampForAge = System.currentTimeMillis();
+  private long age = 0;
 
   public MetricsSink() {
     rms = CompatibilitySingletonFactory.getInstance(MetricsReplicationSource.class);
@@ -47,8 +48,12 @@ public class MetricsSink {
    * @return the age that was set
    */
   public long setAgeOfLastAppliedOp(long timestamp) {
-    lastTimestampForAge = timestamp;
-    long age = System.currentTimeMillis() - lastTimestampForAge;
+    if (lastTimestampForAge != timestamp) {
+      lastTimestampForAge = timestamp;
+      this.age = System.currentTimeMillis() - lastTimestampForAge;
+    } else {
+      this.age = 0;
+    }
     rms.setGauge(SINK_AGE_OF_LAST_APPLIED_OP, age);
     return age;
   }

detail discussion in dev@hbase

Issue Links

• is related to: HBASE-11143 Improve replication metrics