[HBASE-20604] ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.9
• Component/s: Replication, wal
• Assignee: Esteban Gutierrez
• Reporter: Esteban Gutierrez
• Created: 18/May/18 19:21
• Updated: 23/Apr/19 11:51
• Resolved: 08/Nov/18 18:22

Description

Every time we call ProtobufLogReader#readNext we consume the input stream associated to the FSDataInputStream from the WAL that we are reading. Under certain conditions, e.g. when using the encryption at rest (CryptoInputStream) the stream can return partial data which can cause a premature EOF that cause inputStream.getPos() to return to the same origina position causing ProtobufLogReader#readNext to re-try over the reads until the WAL is rolled.

The side effect of this issue is that ReplicationSource can get stuck until the WAL is rolled and causing replication delays up to an hour in some cases.