[HBASE-12522] Backport WAL refactoring to branch-1

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.99.2
• Component/s: wal

Release Note

HBase internals for the write ahead log have been refactored. Advanced users of HBase should be aware of the following changes.

Public Audience
  - The Admin API for asking a region server to roll WAL files has changed from a synchronous command that returns a set of regions the WAL implementation would like flushed into an asynchronous command that returns nothing. Older clients relying on the former behavior will still be able to interact with newer servers, but the response body will always contain an empty list of regions to flush.
  - The shell command "hlog_roll" has been deprecated. Operators should use the "wal_roll" command instead. This command is subject to the changes described above for the Admin API to roll WAL files.
  - The command for analyzing write ahead logs has been renamed from 'hlog' to 'wal'. The old usage is deprecated and will be removed in a future version.
  - Some utility methods in the HBaseTesetingUtility related to testing write-ahead-logs were changed in incompatible ways. No functionality has been removed, but method names and arguments have changed. See the HBaseTestingUtility javadoc for details.
  - The WALPlayer utility has deprecated the configuration keys used for advanced customization. Users should switch to the updated configuration keys. See the usage information on the WALPlayer tool for details.
  - The HLogInputFormat utility class for processing logs with MapReduce has been deprecated and will be removed in a future version. Users should switch to the WALInputFormat.
  - The labeling of server metrics on the region server status pages changed. Previously, the number of backing files for the write ahead log was labeled 'Num. HLog Files'. If you wish to see this statistic now, please look for the label 'Num. WAL Files.' If you rely on JMX for these metrics, their location has not changed.

LimitedPrivate(COPROC) Audience, LimitedPrivate(PHOENIX)
  - The RegionObserver API has been updated. The changes are both binary and source backwards compatible for coprocessors that use the BaseRegionObserver class. For those that implement RegionObserver directly the changes are binary backwards compatible. Depending on the internals of future HBase versions, coprocessors using the deprecated API may not see all WAL related events. Users are strongly encouraged to update their use of the API; see the RegionObserver javadoc for details.
  - Classes related to reading WAL entries (ReaderBase, ProtobufLogReader, SequenceFileLogReader) have changed in a backwards incompatible way. Users who referenced HLog.Reader directly or HLog.Entry will have to update. These changes do not impact compatibility with extant wal files.
  - The WALObserver API has been updated. The changes are both binary and source backwards compatible for coprocessors that use the BaseWALObserver class. For those that implement WALObserver directly the changes are binary backwards compatible. Depending on the internals of future HBase versions, coprocessors using the deprecated API may not see all WAL related events. Users are strongly encouraged to update their use of the API; see the WALObserver javadoc for details.
 - The WALCoprocessorEnvironment has changed in a backwards incompatible way. WALObserver coprocessors that relied on retrieving an object representing the write ahead log instance will have to be updated.

LimitedPrivate(REPLICATION) Audience
 - The WALEntryFilter API has changed in a backwards incompatible way. Implementers will have to be updated.
 - The ReplicationEndpoint.ReplicateContext API has changed in a backwards incompatible way. Implementers who use this interface will have to be updated. These changes do not impact wire compatibility for replicating between clusters.
 - The HLogKey API is deprecated in favor of the WALKey API. Additionally, the HLogKey API has changed in a backwards incompatible way by changing from implementing WriteableComparable<HLogKey> to implementing Writeable and Comparable<WALKey>.

• Assignee: Sean Busbey
• Reporter: Sean Busbey
• Created: 18/Nov/14 20:10
• Updated: 21/Feb/15 23:45
• Resolved: 02/Dec/14 17:22

Description

backport HBASE-10378 to branch-1.

This will let us remove the Deprecated stuff in master, allow some baking time within the 1.x line, and will give us the option of pulling back follow on performance improvements.

Incorporates all master branch patches associated with these jiras:

• HBASE-4744
• HBASE-10378
• HBASE-12532
• HBASE-12535

Also fixes the handling of wals in the NamespaceUpgrade code, originally planned for HBASE-12612.

Issue Links

• depends upon: HBASE-10378 Divide HLog interface into User and Implementor specific interfaces
• depends upon: HBASE-12612 WALFactory use in upgrade path must clean up properly
• depends upon: HBASE-12532 TestFilter failing occasionally with ExitCodeException doing chmod since HBASE-10378
• depends upon: HBASE-12535 NPE in WALFactory close under contention for getInstance
• depends upon: HBASE-4744 Remove @Ignore for testLogRollAfterSplitStart
• is depended upon by: HBASE-12576 Add metrics for rolling the HLog if there are too few DN's in the write pipeline
• is related to: HBASE-12567 Track remaining issues for HBase-1.0