HBase
  1. HBase
  2. HBASE-11236

Last flushed sequence id is ignored by ServerManager

    Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I got lots of error messages like this:

      2014-05-22 08:58:59,793 DEBUG [RpcServer.handler=1,port=20020] master.ServerManager: RegionServer a2428.halxg.cloudera.com,20020,1400742071109 indicates a last flushed sequence id (numberOfStores=9, numberOfStorefiles=2, storefileUncompressedSizeMB=517, storefileSizeMB=517, compressionRatio=1.0000, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=34, totalStaticIndexSizeKB=381, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN) that is less than the previous last flushed sequence id (605446) for region IntegrationTestBigLinkedList, �A��*t�^FU�2��0,1400740489477.a44d3e309b5a7e29355f6faa0d3a4095. Ignoring.

      RegionLoad.toString doesn't print out the last flushed sequence id passed in. Why is it less than the previous one?

        Issue Links

          Activity

          Hide
          Jimmy Xiang added a comment -

          Don't see it any more. Close it.

          Show
          Jimmy Xiang added a comment - Don't see it any more. Close it.
          Hide
          Elliott Clark added a comment -

          I've seen this on 1.2 a decent ammount

          Show
          Elliott Clark added a comment - I've seen this on 1.2 a decent ammount
          Hide
          Pankaj Kumar added a comment -

          I also observed same log in product environment and regions were not opened.

          2015-08-31 19:04:25,448 | WARN  | PriorityRpcServer.handler=13,queue=1,port=21300 | RegionServer *.*.*.*,21302,1441017551749 indicates a last flushed sequence id (53128) that is less than the previous last flushed sequence id (53131) for region hbase:meta,,1 Ignoring. | org.apache.hadoop.hbase.master.ServerManager.updateLastFlushedSequenceIds(ServerManager.java:299)
          
          Show
          Pankaj Kumar added a comment - I also observed same log in product environment and regions were not opened. 2015-08-31 19:04:25,448 | WARN | PriorityRpcServer.handler=13,queue=1,port=21300 | RegionServer *.*.*.*,21302,1441017551749 indicates a last flushed sequence id (53128) that is less than the previous last flushed sequence id (53131) for region hbase:meta,,1 Ignoring. | org.apache.hadoop.hbase.master.ServerManager.updateLastFlushedSequenceIds(ServerManager.java:299)
          Hide
          Duo Zhang added a comment -

          I think this could happen without HBASE-13811 where stack fix a issue that we may report a wrong last flushed sequence id if a flush is aborted.

          Elliott Clark Any more informations? What's happened to the region before this log(reassignment or flush?)? Maybe there are other issues.

          Thanks.

          Show
          Duo Zhang added a comment - I think this could happen without HBASE-13811 where stack fix a issue that we may report a wrong last flushed sequence id if a flush is aborted. Elliott Clark Any more informations? What's happened to the region before this log(reassignment or flush?)? Maybe there are other issues. Thanks.
          Hide
          Pankaj Kumar added a comment -

          Duo Zhang Looks like root cause is different, we have HBASE-13811 in our version. I am also trying to figure out the relevant info from logs.

          Show
          Pankaj Kumar added a comment - Duo Zhang Looks like root cause is different, we have HBASE-13811 in our version. I am also trying to figure out the relevant info from logs.
          Hide
          Stephen Yuan Jiang added a comment -

          Pankaj Kumar, could you provide more information on this? I just wonder whether you suffer data loss on this?

          Show
          Stephen Yuan Jiang added a comment - Pankaj Kumar , could you provide more information on this? I just wonder whether you suffer data loss on this?
          Hide
          Pankaj Kumar added a comment -

          Sorry Stephen Yuan Jiang, couldn't find any useful information that time. Later I didn't see this log again.

          Show
          Pankaj Kumar added a comment - Sorry Stephen Yuan Jiang , couldn't find any useful information that time. Later I didn't see this log again.
          Show
          Allan Yang added a comment - ignore flushed sequence id can cause data loss, please ref to https://issues.apache.org/jira/browse/HBASE-16649?focusedCommentId=15502490&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15502490
          Hide
          Sebastien Barrier added a comment -

          we are having the same issue with hbase-1.2.0+cdh5.8.0+160-1:

          2016-09-22 02:34:21,805 WARN [B.defaultRpcServer.handler=43,queue=3,port=60000] master.ServerManager: RegionServer dr2,60020,1470414843847 indicates a last flushed sequence id (6178551) that is less than the previous last flushed sequence id (9091225) for region user_entry_tags,?,1472495644435.f5cbb00e42bcfd353e2041818b69fd6f. Ignoring.

          Show
          Sebastien Barrier added a comment - we are having the same issue with hbase-1.2.0+cdh5.8.0+160-1: 2016-09-22 02:34:21,805 WARN [B.defaultRpcServer.handler=43,queue=3,port=60000] master.ServerManager: RegionServer dr2,60020,1470414843847 indicates a last flushed sequence id (6178551) that is less than the previous last flushed sequence id (9091225) for region user_entry_tags,?,1472495644435.f5cbb00e42bcfd353e2041818b69fd6f. Ignoring.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jimmy Xiang
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development