XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0, 2.0.0
    • None
    • None
    • Reviewed
    • Hide
      For users on a version of Hadoop that supports tiered storage policies (i.e. Apache Hadoop 2.6.0+), HBase now allows users to opt-in to having the write ahead log placed on the SSD tier. Users on earlier versions of Hadoop will be unable to take advantage of this feature.

      Use of tiered storage is controlled by a new RegionServer config, hbase.wal.storage.policy. It defaults to the value 'NONE', which will rely on HDFS defaults for a policy decision.

      User can specify ONE_SSD or ALL_SSD as the value:
      ONE_SSD: place only one replica of WAL files in SSD and the remaining in default storage
      ALL_SSD: all replica for WAL files are placed on SSD

      See [the HDFS docs on storage policy|http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html]
      Show
      For users on a version of Hadoop that supports tiered storage policies (i.e. Apache Hadoop 2.6.0+), HBase now allows users to opt-in to having the write ahead log placed on the SSD tier. Users on earlier versions of Hadoop will be unable to take advantage of this feature. Use of tiered storage is controlled by a new RegionServer config, hbase.wal.storage.policy. It defaults to the value 'NONE', which will rely on HDFS defaults for a policy decision. User can specify ONE_SSD or ALL_SSD as the value: ONE_SSD: place only one replica of WAL files in SSD and the remaining in default storage ALL_SSD: all replica for WAL files are placed on SSD See [the HDFS docs on storage policy| http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html ]

    Description

      One way to improve data ingestion rate is to make use of Flash storage.
      HDFS is doing the heavy lifting - see HDFS-7228.

      We assume an environment where:
      1. Some servers have a mix of flash, e.g. 2 flash drives and 4 traditional drives.
      2. Some servers have all traditional storage.
      3. RegionServers are deployed on both profiles within one HBase cluster.

      This JIRA allows WAL to be managed on flash in a mixed-profile environment.

      Attachments

        1. 12848-v4.patch
          4 kB
          Ted Yu
        2. 12848-v4.patch
          4 kB
          Ted Yu
        3. 12848-v3.patch
          4 kB
          Ted Yu
        4. 12848-v2.patch
          4 kB
          Ted Yu
        5. 12848-v1.patch
          3 kB
          Ted Yu

        Issue Links

          Activity

            People

              yuzhihong@gmail.com Ted Yu
              yuzhihong@gmail.com Ted Yu
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: