Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14336 Procedure V2 Phase 1 - Procedure Framework and Making DDL Operations fault tolerant
  3. HBASE-13832

Procedure V2: master fail to start due to WALProcedureStore sync failures when HDFS data nodes count is low

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.1.0, 1.2.0, 2.0.0
    • 1.2.0, 1.1.2, 1.3.0, 2.0.0
    • master, proc-v2
    • None

    Description

      when the data node < 3, we got failure in WALProcedureStore#syncLoop() during master start. The failure prevents master to get started.

      2015-05-29 13:27:16,625 ERROR [WALProcedureStoreSyncThread] wal.WALProcedureStore: Sync slot failed, abort.
      java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[10.333.444.555:50010,DS-3c7777ed-93f4-47b6-9c23-1426f7a6acdc,DISK], DatanodeInfoWithStorage[10.222.666.777:50010,DS-f9c983b4-1f10-4d5e-8983-490ece56c772,DISK]],                     original=[DatanodeInfoWithStorage[10.333.444.555:50010,DS-3c7777ed-93f4-47b6-9c23-1426f7a6acdc,DISK], DatanodeInfoWithStorage[10.222.666.777:50010,DS-f9c983b4-1f10-4d5e-8983-    490ece56c772,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'  in its configuration.
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:951)
      

      One proposal is to implement some similar logic as FSHLog: if IOException is thrown during syncLoop in WALProcedureStore#start(), instead of immediate abort, we could try to roll the log and see whether this resolve the issue; if the new log cannot be created or more exception from rolling the log, we then abort.

      Attachments

        1. hbase-13832-test-hang.patch
          5 kB
          Enis Soztutar
        2. HBASE-13832-v0.patch
          18 kB
          Matteo Bertozzi
        3. HBASE-13832-v1.patch
          18 kB
          Matteo Bertozzi
        4. HBASE-13832-v2.patch
          25 kB
          Matteo Bertozzi
        5. hbase-13832-v3.patch
          25 kB
          Enis Soztutar
        6. HBASE-13832-v4.patch
          31 kB
          Matteo Bertozzi
        7. HBASE-13832-v5.patch
          33 kB
          Matteo Bertozzi
        8. HBASE-13832-v6.patch
          34 kB
          Matteo Bertozzi
        9. HDFSPipeline.java
          6 kB
          Matteo Bertozzi

        Activity

          People

            mbertozzi Matteo Bertozzi
            syuanjiang Stephen Yuan Jiang
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: