Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17437

Support specifying a WAL directory outside of the root directory

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      This patch adds support for specifying a WAL directory outside of the HBase root directory.

      Multiple configuration variables were added to accomplish this:
      hbase.wal.dir: used to configure where the root WAL directory is located. Could be on a different FileSystem than the root directory. WAL directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset.

      hbase.rootdir.perms: Configures FileSystem permissions to set on the root directory. This is '700' by default.

      hbase.wal.dir.perms: Configures FileSystem permissions to set on the WAL directory FileSystem. This is '700' by default.
      Show
      This patch adds support for specifying a WAL directory outside of the HBase root directory. Multiple configuration variables were added to accomplish this: hbase.wal.dir: used to configure where the root WAL directory is located. Could be on a different FileSystem than the root directory. WAL directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset. hbase.rootdir.perms: Configures FileSystem permissions to set on the root directory. This is '700' by default. hbase.wal.dir.perms: Configures FileSystem permissions to set on the WAL directory FileSystem. This is '700' by default.
    • Patch

    Description

      Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some FileSystems (such as Amazon S3) don’t support append or consistent writes. These two properties are imperative for the WAL in order to avoid loss of writes. However, StoreFiles don’t necessarily need the same consistency guarantees (since writes are cached locally and if writes fail, they can always be replayed from the WAL).

      This JIRA aims to allow users to configure a log directory (for WALs) that is outside of the root directory or even in a different FileSystem. The default value will still put the log directory under the root directory.

      Attachments

        1. HBASE-17437.branch-1.001.patch
          104 kB
          Zach York
        2. HBASE-17437.branch-1.002.patch
          128 kB
          Zach York
        3. HBASE-17437.branch-1.003.patch
          128 kB
          Zach York
        4. HBASE-17437.branch-1.004.patch
          128 kB
          Zach York
        5. HBASE-17437.master.001.patch
          94 kB
          Zach York
        6. HBASE-17437.master.002.patch
          94 kB
          Zach York
        7. HBASE-17437.master.003.patch
          94 kB
          Zach York
        8. HBASE-17437.master.004.patch
          94 kB
          Zach York
        9. HBASE-17437.master.005.patch
          103 kB
          Zach York
        10. HBASE-17437.master.006.patch
          102 kB
          Zach York
        11. HBASE-17437.master.007.patch
          102 kB
          Zach York
        12. HBASE-17437.master.008.patch
          100 kB
          Zach York
        13. HBASE-17437.master.009.patch
          100 kB
          Zach York
        14. HBASE-17437.master.010.patch
          100 kB
          Zach York
        15. HBASE-17437.master.011.patch
          96 kB
          Zach York
        16. HBASE-17437.master.012.patch
          95 kB
          Zach York
        17. hbase-17437-branch-1.2.patch
          134 kB
          Yishan Yang
        18. hbase-17437-master.patch
          74 kB
          Yishan Yang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zyork Zach York
            yangyishan0901m Yishan Yang
            Votes:
            0 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment