[HBASE-20691] Storage policy should allow deferring to HDFS

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0, 1.5.0, 2.0.2
• Component/s: Filesystem Integration, wal
• Hadoop Flags: Incompatible change, Reviewed

Release Note

After HBASE-20691 we have changed the default setting of hbase.wal.storage.policy from "HOT" back to "NONE" which means we defer the policy to HDFS. This fixes the problem of release 2.0.0 that the storage policy of WAL directory will defer to HDFS and may not be "HOT" even if you explicitly set hbase.wal.storage.policy to "HOT"

• Assignee: Yu Li
• Reporter: Sean Busbey
• Created: 06/Jun/18 19:12
• Updated: 01/Jul/22 20:23
• Resolved: 05/Jul/18 07:31

Description

In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using "NONE" as the storage policy in hbase configs.

As described on this dev@hbase thread "WAL storage policies and interactions with Hadoop admin tools." we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). Additionally, we can't set the policy to HOT in the event that HDFS has changed the policy for a parent directory of our WALs.

We should put back that ability. Presuming this is done by re-adopting the "NONE" placeholder variable, we need to ensure that value doesn't get passed to HDFS APIs. Since it isn't a valid storage policy attempting to use it will cause a bunch of logging churn (which will be a regression of the problem HBASE-18118 sought to fix).

Issue Links

• Dependent: HBASE-20838 Include hbase-server in precommit test if CommonFSUtils is changed
• is related to: HBASE-12848 Utilize Flash storage for WAL
• is related to: HBASE-14061 Support CF-level Storage Policy
• is related to: HBASE-18118 Default storage policy if not configured cannot be "NONE"
• is related to: HBASE-19858 Backport HBASE-14061 (Support CF-level Storage Policy) to branch-1
• links to: ReviewBoard