XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      For users on a version of Hadoop that supports tiered storage policies (i.e. Apache Hadoop 2.6.0+), HBase now allows users to opt-in to having the write ahead log placed on the SSD tier. Users on earlier versions of Hadoop will be unable to take advantage of this feature.

      Use of tiered storage is controlled by a new RegionServer config, hbase.wal.storage.policy. It defaults to the value 'NONE', which will rely on HDFS defaults for a policy decision.

      User can specify ONE_SSD or ALL_SSD as the value:
      ONE_SSD: place only one replica of WAL files in SSD and the remaining in default storage
      ALL_SSD: all replica for WAL files are placed on SSD

      See [the HDFS docs on storage policy|http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html]
      Show
      For users on a version of Hadoop that supports tiered storage policies (i.e. Apache Hadoop 2.6.0+), HBase now allows users to opt-in to having the write ahead log placed on the SSD tier. Users on earlier versions of Hadoop will be unable to take advantage of this feature. Use of tiered storage is controlled by a new RegionServer config, hbase.wal.storage.policy. It defaults to the value 'NONE', which will rely on HDFS defaults for a policy decision. User can specify ONE_SSD or ALL_SSD as the value: ONE_SSD: place only one replica of WAL files in SSD and the remaining in default storage ALL_SSD: all replica for WAL files are placed on SSD See [the HDFS docs on storage policy| http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html ]

      Description

      One way to improve data ingestion rate is to make use of Flash storage.
      HDFS is doing the heavy lifting - see HDFS-7228.

      We assume an environment where:
      1. Some servers have a mix of flash, e.g. 2 flash drives and 4 traditional drives.
      2. Some servers have all traditional storage.
      3. RegionServers are deployed on both profiles within one HBase cluster.

      This JIRA allows WAL to be managed on flash in a mixed-profile environment.

        Attachments

        1. 12848-v4.patch
          4 kB
          Ted Yu
        2. 12848-v4.patch
          4 kB
          Ted Yu
        3. 12848-v3.patch
          4 kB
          Ted Yu
        4. 12848-v2.patch
          4 kB
          Ted Yu
        5. 12848-v1.patch
          3 kB
          Ted Yu

          Issue Links

            Activity

              People

              • Assignee:
                yuzhihong@gmail.com Ted Yu
                Reporter:
                yuzhihong@gmail.com Ted Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: