[HDFS-11292] log lastWrittenTxId etc info in logSyncAll

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha2
• Component/s: hdfs
• Hadoop Flags: Reviewed
• Assignee: Yongjun Zhang
• Reporter: Yongjun Zhang
• Created: 04/Jan/17 19:03
• Updated: 02/Oct/19 17:14
• Resolved: 10/Jan/17 00:33

Description

For the issue reported in HDFS-10943, even after HDFS-7964's fix is included, the problem still exists, this means there might be some synchronization issue.

To diagnose that, create this jira to report the lastWrittenTxId info in logSyncAll() call, such that we can compare against the error message reported in HDFS-7964

Specifically, there is two possibility for the HDFS-10943 issue:

1. logSyncAll() (statement A in the code quoted below) doesn't flush all requested txs for some reason

2. logSyncAll() does flush all requested txs, but some new txs sneaked in between A and B. It's observed that the lastWrittenTxId in B and C are the same.

This proposed reporting would help confirming if 2 is true.

 public synchronized void endCurrentLogSegment(boolean writeEndTxn) {
    LOG.info("Ending log segment " + curSegmentTxId);
    Preconditions.checkState(isSegmentOpen(),
        "Bad state: %s", state);

    if (writeEndTxn) {
      logEdit(LogSegmentOp.getInstance(cache.get(),
          FSEditLogOpCodes.OP_END_LOG_SEGMENT));
    }
    // always sync to ensure all edits are flushed.
A.    logSyncAll();

B.    printStatistics(true);

    final long lastTxId = getLastWrittenTxId();

    try {
C.      journalSet.finalizeLogSegment(curSegmentTxId, lastTxId);
      editLogStream = null;
    } catch (IOException e) {
      //All journals have failed, it will be handled in logSync.
    }

    state = State.BETWEEN_LOG_SEGMENTS;
  }

Issue Links

• is related to: HDFS-10943 rollEditLog expects empty EditsDoubleBuffer.bufCurrent which is not guaranteed
• is related to: HDFS-11306 Print remaining edit logs from buffer if edit log can't be rolled.