[HBASE-8670] [0.94] Backport HBASE-8449,HBASE-8204 and HBASE-8699 to 0.94 (Refactor recoverLease retries and pauses)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.94.11
• Component/s: Filesystem Integration, master, wal
• Hadoop Flags: Reviewed
• Assignee: Enis Soztutar
• Reporter: Enis Soztutar
• Created: 31/May/13 21:54
• Updated: 21/Aug/13 00:08
• Resolved: 02/Aug/13 05:21

Description

Some history:
Up until 0.94.8, Hbase did not check the result of recoverLease() call, but things kind of worked since we are checking for 0-length files in distributed log split tasks from region servers. If lease recovery is not finished, the log file will report 0 length, and the task will fail, and master will then re-call recoverLease() and reassign the task. This scheme might fail for log files that are larger than 1 hdfs block though.
In 0.94.8, we committed (HBASE-8354, which is backport of HBASE-7878) and later increased the sleep time to 4 secs in HBASE-8389.
However, the proper solution arrived in trunk in HBASE-8449 which uses a backoff sleep policy + isFileClosed() api. We should backport this patch to 0.94 as well.

isFileClosed() is released in Hadoop 1.2.0 (HDFS-4774) and 2.0.5(HDFS-4525).

Issue Links

• relates to: HDFS-4525 Provide an API for knowing that whether file is closed or not.
• relates to: HBASE-7878 recoverFileLease does not check return value of recoverLease
• relates to: HBASE-8389 HBASE-8354 forces Namenode into loop with lease recovery requests
• relates to: HBASE-8449 Refactor recoverLease retries and pauses informed by findings over in hbase-8389
• relates to: HBASE-8204 Don't use hdfs append during lease recovery
• relates to: HBASE-8354 Backport HBASE-7878 'recoverFileLease does not check return value of recoverLease' to 0.94
• relates to: HBASE-8699 Parameter to DistributedFileSystem#isFileClosed should be of type Path
• relates to: HDFS-4774 Backport HDFS-4525 'Provide an API for knowing whether file is closed or not' to branch-1