Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11099

Two situations where we could open a region with smaller sequence number

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0, 0.99.1, 2.0.0
    • 0.98.9, 0.99.2
    • regionserver
    • None
    • Reviewed

    Description

      Recently I happened to run into code where we potentially could open region with smaller sequence number:

      1) Inside function: HRegion#internalFlushcache. This is due to we change the way WAL Sync where we use late binding(assign sequence number right before wal sync).
      The flushSeqId may less than the change sequence number included in the flush which may cause later region opening code to use a smaller than expected sequence number when we reopen the region.

      flushSeqId = this.sequenceId.incrementAndGet();
      ...
      mvcc.waitForRead(w);
      

      2) HRegion#replayRecoveredEdits where we have following code:

      ...
                if (coprocessorHost != null) {
                  status.setStatus("Running pre-WAL-restore hook in coprocessors");
                  if (coprocessorHost.preWALRestore(this.getRegionInfo(), key, val)) {
                    // if bypass this log entry, ignore it ...
                    continue;
                  }
                }
      ...
                currentEditSeqId = key.getLogSeqNum();
      

      If coprocessor skip some tail WALEdits, then the function will return smaller currentEditSeqId. In the end, a region may also open with a smaller sequence number. This may cause data loss because Master may record a larger flushed sequence Id and some WALEdits maybe skipped during recovery if the region fail again.

      Attachments

        1. HBASE-11099.v1-0.98.patch
          2 kB
          Stephen Yuan Jiang
        2. HBASE-11099.v1-2.0.patch
          2 kB
          Stephen Yuan Jiang

        Issue Links

          Activity

            People

              syuanjiang Stephen Yuan Jiang
              jeffreyz Jeffrey Zhong
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: