[HBASE-27632] Refactor WAL.Reader implementation so we can better support WAL splitting and replication

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-alpha-4
• Component/s: Replication, wal
• Hadoop Flags: Reviewed

Release Note

Introduced two types of WAL reader, WALStreamReader and WALTailingReader. The former one is for most use cases where we only need to read closed WAL files, such as splitting and replaying, and the second one is for replication, where we need to tail a WAL file which is being written currently.

The config 'hbase.regionserver.hlog.reader.impl' is removed, as now we have two types of WAL reader. And starting from at least 0.98, we do not support sequence file format WAL file any more, the only accepted format is protobuf. And for encrypted WAL files, now you do not need to specify SecureProtobufLogReader either, as the new implementation will choose to use SecureWALCellCodec based on the properties in WALHeader.
There is still a config called 'hbase.regionserver.wal.stream.reader.impl', for specifying the implementation for WALStreamReader. This is only designed to be used in HBase itself, for injecting some error when writing UTs, downstream users should not use it as it is unstable.

• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 10/Feb/23 07:32
• Updated: 20/Mar/23 04:16
• Resolved: 10/Mar/23 15:04

Description

WAL splitting and replication has completely different requirements on WAL reader, the latter one is much comlicated as it wants to tail a WAL file which is currently being written.
I think split them into two classes is better here.

But we exposes two configurations out

'hbase.regionserver.hlog.reader.impl'
'hbase.regionserver.hlog.writer.impl'

which indicates that we only have one implementation for wal reader.

These two configurations are inrtoduced in 0.96, for reading the pre 0.94 sequence file format WAL file. And starting from 2.0.0, we have already dropped the support of reading sequence file format WAL file, so I think the configurations are useless now.
Now we stil require users to specify SecureProtobufLogReader and SecureProtobufLogWriter when enable WAL encryption, but actually they are both the only choice, so just use 'hbase.regionserver.wal.encryption' to determine whether we need to use SecureProtobufLogWriter to write WAL. And for reading, I do not think we should rely on the configuration, we should detect whether the WAL file itself is encrypted(I believe we have already done this in SecureProtobufLogReader.readHeader.

So in general, I prefer we just mark these two configurations deprecated and do not use them in our code base, so we can have different Reader implementations for WAL splitting and replication.

Issue Links

• breaks: HBASE-27709 Fix UT failures after HBASE-27632 goes in
• contains: HBASE-27677 ReplaySyncReplicationWALCallable should consider WAL files which are not closed cleanly
• is related to: HBASE-27621 Also clear the Dictionary when resetting when reading compressed WAL file
• relates to: HBASE-27702 Remove 'hbase.regionserver.hlog.writer.impl' config
• relates to: HBASE-27722 Update documentation about how to enable encryption on WAL
• links to: GitHub Pull Request #5055