[HBASE-24625] AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.0.0, 2.2.0, 2.3.0
• Fix Version/s: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6
• Component/s: Replication, wal
• Hadoop Flags: Reviewed

Release Note

We add a method getSyncedLength in WALProvider.WriterBase interface for WALFileLengthProvider used for replication, considering the case if we use AsyncFSWAL,we write to 3 DNs concurrently,according to the visibility guarantee of HDFS, the data will be available immediately
when arriving at DN since all the DNs will be considered as the last one in pipeline.This means replication may read uncommitted data and replicate it to the remote cluster and cause data inconsistency.The method WriterBase#getLength may return length which just in hdfs client buffer and not successfully synced to HDFS, so we use this method WriterBase#getSyncedLength to return the length successfully synced to HDFS and replication thread could only read writing WAL file limited by this length.
see also HBASE-14004 and this document for more details:
https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit#

Before this patch, replication may read uncommitted data and replicate it to the slave cluster and cause data inconsistency between master and slave cluster, we could use FSHLog instead of AsyncFSWAL to reduce probability of inconsistency without this patch applied.
    

• Assignee: chenglei
• Reporter: chenglei
• Created: 24/Jun/20 13:41
• Updated: 11/Aug/20 00:38
• Resolved: 10/Aug/20 02:23

Description

By HBASE-14004, we introduce WALFileLengthProvider interface to keep the current writing wal file length by ourselves, WALEntryStream used by ReplicationSourceWALReader could only read WAL file byte size <= WALFileLengthProvider.getLogFileSizeIfBeingWritten if the WAL file is current been writing on the same RegionServer .

AsyncFSWAL implements WALFileLengthProvider by AbstractFSWAL.getLogFileSizeIfBeingWritten, just as folllows :

   public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
    rollWriterLock.lock();
    try {
      Path currentPath = getOldPath();
      if (path.equals(currentPath)) {
        W writer = this.writer;
        return writer != null ? OptionalLong.of(writer.getLength()) : OptionalLong.empty();
      } else {
        return OptionalLong.empty();
      }
    } finally {
      rollWriterLock.unlock();
    }
  }

For AsyncFSWAL, above AsyncFSWAL.writer is AsyncProtobufLogWriter ，and AsyncProtobufLogWriter.getLength is as follows:

    public long getLength() {
        return length.get();
    }

But for AsyncProtobufLogWriter, any append method may increase the above AsyncProtobufLogWriter.length, especially for following AsyncFSWAL.append
method just appending the WALEntry to FanOutOneBlockAsyncDFSOutput.buf:

     public void append(Entry entry) {
          int buffered = output.buffered();
          try {
              entry.getKey().
              getBuilder(compressor).setFollowingKvCount(entry.getEdit().size()).build()
              .writeDelimitedTo(asyncOutputWrapper);
          } catch (IOException e) {
             throw new AssertionError("should not happen", e);
          }
   
        try {
           for (Cell cell : entry.getEdit().getCells()) {
             cellEncoder.write(cell);
           }
          } catch (IOException e) {
           throw new AssertionError("should not happen", e);
         }
         length.addAndGet(output.buffered() - buffered);
     }

That is to say, AsyncFSWAL.getLogFileSizeIfBeingWritten could not reflect the file length which successfully synced to underlying HDFS, which is not as expected.

Issue Links

• is related to: HBASE-24691 Fix flaky TestWALEntryStream
• relates to: HBASE-14004 [Replication] Inconsistency between Memstore and WAL may result in data in remote cluster that is not in the origin
• links to: GitHub Pull Request #1970
• links to: GitHub Pull Request #2028
• links to: GitHub Pull Request #2034
• links to: GitHub Pull Request #2055