Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5699

Run with > 1 WAL in HRegionServer

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 1.0.0, 1.1.0
    • Performance, wal
    • None
    • Hide
      HBase's write-ahead-log (WAL) can now be configured to use multiple HDFS pipelines in parallel to provide better write throughput for clusters by using additional disks. By default, HBase will still use only a single HDFS-based WAL.

      To run with multiple WALs, alter the hbase-site.xml property "hbase.wal.provider" to have the value "multiwal". To return to having HBase determine what kind of WAL implementation to use either remove the property all together or set it to "defaultProvider".

      Altering the WAL provider used by a particular RegionServer requires restarting that instance. RegionServers using the original WAL implementation and those using the "multiwal" implementation can each handle recovery of either set of WALs, so a zero-downtime configuration update is possible through a rolling restart.

      This issue introduces the following configurations:

      hbase.wal.regiongrouping.numgroups is how many provider instances the 'multiwal' should create. Default is two.

      hbase.wal.regiongrouping.strategy is the strategy to use figuring which provider instance an edit should go to. Default is 'identity'. Identity is the only current built-in option.

      hbase.wal.regiongrouping.delegate is the type of the provider the multiwal creates. Default is default from WALFactory.
      Show
      HBase's write-ahead-log (WAL) can now be configured to use multiple HDFS pipelines in parallel to provide better write throughput for clusters by using additional disks. By default, HBase will still use only a single HDFS-based WAL. To run with multiple WALs, alter the hbase-site.xml property "hbase.wal.provider" to have the value "multiwal". To return to having HBase determine what kind of WAL implementation to use either remove the property all together or set it to "defaultProvider". Altering the WAL provider used by a particular RegionServer requires restarting that instance. RegionServers using the original WAL implementation and those using the "multiwal" implementation can each handle recovery of either set of WALs, so a zero-downtime configuration update is possible through a rolling restart. This issue introduces the following configurations: hbase.wal.regiongrouping.numgroups is how many provider instances the 'multiwal' should create. Default is two. hbase.wal.regiongrouping.strategy is the strategy to use figuring which provider instance an edit should go to. Default is 'identity'. Identity is the only current built-in option. hbase.wal.regiongrouping.delegate is the type of the provider the multiwal creates. Default is default from WALFactory.

    Attachments

      Issue Links

        depends upon

        Sub-task - The sub-task of the issue HBASE-10378 Divide HLog interface into User and Implementor specific interfaces

        • Major - Major loss of function.
        • Closed
        is depended upon by

        Umbrella - An overarching type made of sub-tasks HBASE-8338 Latency Resilience; umbrella list of issues that will help us ride over bad disk, bad region, ec2, etc.

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed
        is duplicated by

        New Feature - A new feature of the product, which has yet to be developed. HBASE-6981 multiple commit logs per region server

        • Major - Major loss of function.
        • Closed
        is related to

        Sub-task - The sub-task of the issue HBASE-5937 Refactor HLog into an interface.

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HBASE-6116 Allow parallel HDFS writes for HLogs.

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. HBASE-11270 Separate replication table's hlog from nonreplication table

        • Major - Major loss of function.
        • Closed

        Umbrella - An overarching type made of sub-tasks HBASE-14457 Umbrella: Improve Multiple WAL for production usage

        • Major - Major loss of function.
        • Closed
        relates to

        New Feature - A new feature of the product, which has yet to be developed. HBASE-10278 Provide better write predictability

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. ACCUMULO-1083 add concurrency to HDFS write-ahead log

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HBASE-8610 Introduce interfaces to support MultiWAL

        • Major - Major loss of function.
        • Closed
        links to

        Web Link reviewboard

        Activity

          People

            busbey Sean Busbey
            binlijin Lijin Bin
            Votes:
            0 Vote for this issue
            Watchers:
            62 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: