[HBASE-16077] Replication status doesnt show failed RS metrics in CLI

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Assignee: Unassigned
• Reporter: Bibin Chundatt
• Created: 21/Jun/16 05:45
• Updated: 20/May/19 11:52
• Resolved: 20/May/19 11:52

Description

Steps to reproduce
================

1. Create 2 clusters and configure replication
2. Create TABLE 1 and enable table replication
3. Shutdown Cluster 2 for short period.
4. Load data to TABLE 1
5. Shutdown Region Server whr Region of TABLE 1 is available
6. Check metrics using CLI

hbase(main):003:0* status 'replication'
2016-06-14 00:58:04,664 INFO  [main] ipc.AbstractRpcClient: RPC Server Kerberos principal name for service=MasterService is hbase/[e-mail]
version 1.0.2
3 live servers
    host-10-19-92-200:
       SOURCE: PeerID=11, SizeOfLogQueue=0, ShippedBatches=30, ShippedOps=1351, ShippedBytes=1513127672, LogReadInBytes=662648911, LogEditsRead=1546, LogEditsFiltered=1409, SizeOfLogToReplicate=0, TimeWillBeTakenForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0, AgeOfLastShippedOp=0, TimeStampsOfLastShippedOp=Tue Jun 14 00:58:01 IST 2016, Replication Lag=0
       SINK  : AppliedBatches=2, AppliedOps=5, AppliedHFiles=3, AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Mon Jun 13 02:18:06 IST 2016
    host-10-19-92-187:
       SOURCE: PeerID=11, SizeOfLogQueue=0, ShippedBatches=0, ShippedOps=0, ShippedBytes=0, LogReadInBytes=65719, LogEditsRead=112, LogEditsFiltered=112, SizeOfLogToReplicate=0, TimeWillBeTakenForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0, AgeOfLastShippedOp=0, TimeStampsOfLastShippedOp=Tue Jun 14 00:58:01 IST 2016, Replication Lag=0
       SINK  : AppliedBatches=0, AppliedOps=0, AppliedHFiles=0, AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Mon Jun 13 09:07:20 IST 2016
    host-10-19-92-188:
       SOURCE: PeerID=11, SizeOfLogQueue=0, ShippedBatches=39, ShippedOps=1730, ShippedBytes=1937609744, LogReadInBytes=848439638, LogEditsRead=1671, LogEditsFiltered=1497, SizeOfLogToReplicate=0, TimeWillBeTakenForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0, AgeOfLastShippedOp=0, TimeStampsOfLastShippedOp=Tue Jun 14 00:58:03 IST 2016, Replication Lag=0
       SINK  : AppliedBatches=1, AppliedOps=1, AppliedHFiles=0, AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Mon Jun 13 01:53:53 IST 2016

JMX output

{
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
    "modelerType" : "RegionServer,sub=Replication",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "host-10-19-92-200",
    "source.11.sizeOfLogToReplicate" : 537,
    "source.11-host-10-19-92-187,21302,1465787242095.sizeOfLogToReplicate" : 282766680,
    "source.shippedHFiles" : 0,
    "source.ageOfLastShippedOp" : 0,
    "source.11.shippedHFiles" : 0,
    "source.11-host-10-19-92-187,21302,1465787242095.ageOfLastShippedOp" : 0,
    "source.shippedKBs" : 1477663,
    "source.sizeOfHFileRefsQueue" : 0,
    "source.logReadInBytes" : 691148656,
    "source.11-host-10-19-92-187,21302,1465787242095.logEditsRead" : 39,
    "source.11-host-10-19-92-187,21302,1465787242095.shippedOps" : 0,
    "source.11.logEditsFiltered" : 1244,
    "source.sizeOfLogQueue" : 4,
    "source.timeWillBeTakenForLogToReplicate" : 1,
    "sink.ageOfLastAppliedOp" : 0,
    "source.11-host-10-19-92-187,21302,1465787242095.timeWillBeTakenForLogToReplicate" : 0,
    "source.logEditsRead" : 1420,
    "source.11.sizeOfLogQueue" : 0,
    "source.11-host-10-19-92-187,21302,1465787242095.logEditsFiltered" : 32,
    "source.11-host-10-19-92-187,21302,1465787242095.shippedHFiles" : 0,
    "source.shippedOps" : 1351,
    "source.11.shippedKBs" : 1477663,
    "source.11.logReadInBytes" : 662562515,
    "sink.appliedHFiles" : 3,
    "source.11.sizeOfHFileRefsQueue" : 0,
    "source.logEditsFiltered" : 1276,
    "source.shippedBytes" : 1513127672,
    "source.11-host-10-19-92-187,21302,1465787242095.shippedBatches" : 0,
    "source.11.shippedBytes" : 1513127672,
    "sink.appliedOps" : 5,
    "source.11-host-10-19-92-187,21302,1465787242095.sizeOfLogQueue" : 4,
    "source.11.shippedBatches" : 30,
    "source.11-host-10-19-92-187,21302,1465787242095.sizeOfHFileRefsQueue" : 0,
    "source.11.timeWillBeTakenForLogToReplicate" : 1,
    "source.11-host-10-19-92-187,21302,1465787242095.logReadInBytes" : 28586141,
    "source.11.shippedOps" : 1351,
    "source.shippedBatches" : 30,
    "source.11-host-10-19-92-187,21302,1465787242095.shippedKBs" : 0,
    "source.sizeOfLogToReplicate" : 537,
    "source.11.ageOfLastShippedOp" : 0,
    "sink.appliedBatches" : 2,
    "source.11.logEditsRead" : 1381,
    "source.11-host-10-19-92-187,21302,1465787242095.shippedBytes" : 0
  }

source.11-host-10-19-92-187,21302,1465787242095 not available in CLI