Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19024

Configurable default durability for synchronous WAL

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.1.0, 1.5.0
    • Component/s: wal
    • Labels:
      None
    • Release Note:
      Hide
      The default durability setting for the synchronous WAL is Durability.SYNC_WAL, which triggers HDFS hflush() to flush edits to the datanodes. We also support Durability.FSYNC_WAL, which instead triggers HDFS hsync() to flush _and_ fsync edits. This change introduces the new configuration setting "hbase.wal.hsync", defaulting to FALSE, that if set to TRUE changes the default durability setting for the synchronous WAL to FSYNC_WAL.
      Show
      The default durability setting for the synchronous WAL is Durability.SYNC_WAL, which triggers HDFS hflush() to flush edits to the datanodes. We also support Durability.FSYNC_WAL, which instead triggers HDFS hsync() to flush _and_ fsync edits. This change introduces the new configuration setting "hbase.wal.hsync", defaulting to FALSE, that if set to TRUE changes the default durability setting for the synchronous WAL to FSYNC_WAL.

      Description

      At present we do not have an option to hsync WAL edits to the disk for better durability. In our local tests we see 10-15% latency impact of using hsync instead of hflush which is not very high.
      We should have a configurable option to hysnc WAL edits instead of just sync/hflush which will call the corresponding API on the hadoop side. Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling FSDataOutputStream sync/hflush on the hadoop side. This can be modified to let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can keep the default value to sync as the current behavior and hsync can be enabled based on explicit configuration.

        Attachments

        1. HBASE-19024-master.v10.patch
          31 kB
          Andrew Purtell
        2. HBASE-19024.branch-1.2.005.patch
          4 kB
          stack
        3. HBASE-19024.branch-1.2.004.patch
          3 kB
          stack
        4. HBASE-19024.branch-1.2.003.patch
          3 kB
          stack
        5. HBASE-19024.branch-1.2.002.patch
          0.8 kB
          stack
        6. HBASE-19024.branch-1.2.001.patch
          0.8 kB
          stack
        7. master.v9.patch
          29 kB
          Harshal Jain
        8. master.v6.patch
          21 kB
          Harshal Jain
        9. master.v5.patch
          21 kB
          Harshal Jain
        10. branch-1.v1.branch-1.patch
          25 kB
          Harshal Jain
        11. master.v5.patch
          21 kB
          Harshal Jain
        12. master.v3.patch
          22 kB
          Harshal Jain
        13. branch-1.branch-1.patch
          25 kB
          Harshal Jain
        14. master.v2.patch
          21 kB
          Harshal Jain
        15. master.patch
          13 kB
          Harshal Jain

          Issue Links

            Activity

              People

              • Assignee:
                harshaldjain Harshal Jain
                Reporter:
                vik.karma Vikas Vishwakarma
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: