Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20691

Storage policy should allow deferring to HDFS

    Details

    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      After HBASE-20691 we have changed the default setting of hbase.wal.storage.policy from "HOT" back to "NONE" which means we defer the policy to HDFS. This fixes the problem of release 2.0.0 that the storage policy of WAL directory will defer to HDFS and may not be "HOT" even if you explicitly set hbase.wal.storage.policy to "HOT"
      Show
      After HBASE-20691 we have changed the default setting of hbase.wal.storage.policy from "HOT" back to "NONE" which means we defer the policy to HDFS. This fixes the problem of release 2.0.0 that the storage policy of WAL directory will defer to HDFS and may not be "HOT" even if you explicitly set hbase.wal.storage.policy to "HOT"

      Description

      In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using "NONE" as the storage policy in hbase configs.

      As described on this dev@hbase thread "WAL storage policies and interactions with Hadoop admin tools." we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). Additionally, we can't set the policy to HOT in the event that HDFS has changed the policy for a parent directory of our WALs.

      We should put back that ability. Presuming this is done by re-adopting the "NONE" placeholder variable, we need to ensure that value doesn't get passed to HDFS APIs. Since it isn't a valid storage policy attempting to use it will cause a bunch of logging churn (which will be a regression of the problem HBASE-18118 sought to fix).

        Attachments

        1. HBASE-20691.patch
          1 kB
          Yu Li
        2. HBASE-20691.v2.patch
          4 kB
          Yu Li
        3. HBASE-20691.v3.patch
          6 kB
          Yu Li
        4. HBASE-20691.v4.patch
          11 kB
          Yu Li
        5. HBASE-20691.v5.patch
          14 kB
          Yu Li
        6. HBASE-20691.v6.patch
          14 kB
          Yu Li
        7. HBASE-20691.v7.patch
          14 kB
          Yu Li
        8. HBASE-20691.v8.patch
          14 kB
          Yu Li
        9. HBASE-20691.branch-1.patch
          14 kB
          Yu Li
        10. HBASE-20691.branch-1.v2.patch
          14 kB
          Yu Li
        11. HBASE-20691.branch-1.v2.patch
          14 kB
          Yu Li

          Issue Links

            Activity

              People

              • Assignee:
                carp84 Yu Li
                Reporter:
                busbey Sean Busbey
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: