Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19024

Configurable default durability for synchronous WAL

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.1.0, 1.5.0
    • Component/s: wal
    • Labels:
      None
    • Release Note:
      Hide
      The default durability setting for the synchronous WAL is Durability.SYNC_WAL, which triggers HDFS hflush() to flush edits to the datanodes. We also support Durability.FSYNC_WAL, which instead triggers HDFS hsync() to flush _and_ fsync edits. This change introduces the new configuration setting "hbase.wal.hsync", defaulting to FALSE, that if set to TRUE changes the default durability setting for the synchronous WAL to FSYNC_WAL.
      Show
      The default durability setting for the synchronous WAL is Durability.SYNC_WAL, which triggers HDFS hflush() to flush edits to the datanodes. We also support Durability.FSYNC_WAL, which instead triggers HDFS hsync() to flush _and_ fsync edits. This change introduces the new configuration setting "hbase.wal.hsync", defaulting to FALSE, that if set to TRUE changes the default durability setting for the synchronous WAL to FSYNC_WAL.

      Description

      At present we do not have an option to hsync WAL edits to the disk for better durability. In our local tests we see 10-15% latency impact of using hsync instead of hflush which is not very high.
      We should have a configurable option to hysnc WAL edits instead of just sync/hflush which will call the corresponding API on the hadoop side. Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling FSDataOutputStream sync/hflush on the hadoop side. This can be modified to let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can keep the default value to sync as the current behavior and hsync can be enabled based on explicit configuration.

        Attachments

        1. branch-1.branch-1.patch
          25 kB
          Harshal Jain
        2. branch-1.v1.branch-1.patch
          25 kB
          Harshal Jain
        3. HBASE-19024.branch-1.2.001.patch
          0.8 kB
          Michael Stack
        4. HBASE-19024.branch-1.2.002.patch
          0.8 kB
          Michael Stack
        5. HBASE-19024.branch-1.2.003.patch
          3 kB
          Michael Stack
        6. HBASE-19024.branch-1.2.004.patch
          3 kB
          Michael Stack
        7. HBASE-19024.branch-1.2.005.patch
          4 kB
          Michael Stack
        8. HBASE-19024-master.v10.patch
          31 kB
          Andrew Kyle Purtell
        9. master.patch
          13 kB
          Harshal Jain
        10. master.v2.patch
          21 kB
          Harshal Jain
        11. master.v3.patch
          22 kB
          Harshal Jain
        12. master.v5.patch
          21 kB
          Harshal Jain
        13. master.v5.patch
          21 kB
          Harshal Jain
        14. master.v6.patch
          21 kB
          Harshal Jain
        15. master.v9.patch
          29 kB
          Harshal Jain

          Issue Links

            Activity

              People

              • Assignee:
                harshaldjain Harshal Jain
                Reporter:
                vik.karma Vikas Vishwakarma
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: