1. HBase
  2. HBASE-11218

Data loss in HBase standalone mode


    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:
    • Release Note:
      Fix for local filesystem data loss in standalone and pseudo-distributed mode.


      Data loss in HBase standalone mode.
      How to produce it

      1. Start HBase standalone mode.
      2. Create a table using hbase shell.
      3. Scan '.META.' and you will find data in meta table
      4. Kill the HBase process with -9 option
      5. Start the HBase agaion
      6. Scan '.META.' and you will find nothing in meta table.

      There are three main reasons.

      1. FSDataOutputStream.sync should call flush() if the underlying wrapped stream is not Syncable. See HADOOP-8861
      2. writeChecksum is ture in default LocalFileSystem and the ChecksumFSOutputSummer will buffer the data, which make the waledits are not written to os's filesystem with sync method immediately, and those edits will be lost in regionserver's failover.
      3. The MiniZooKeeperCluster deletes the old zk data at startup which maye cause data loss in meta table. The failover procedure is: split pre root regionserver's hlog -> assign root -> split pre meta regionserver's hlog -> assign meta -> split all other regionservers' hlogs -> assign other regions. If there is no data in zookeeper, we will get null for root regionserver and then assign root table. Some data in root table maybe be lost for some root's WalEdits have not been splited and replayed. So does the Meta table.

      I finished the patch for 0.94 and am working on the patch for trunk. Suggestions are welcomed.

      1. HBASE-11218-trunk-v1.diff
        7 kB
        Liu Shaohui
      2. HBASE-11218-trunk-v2.diff
        7 kB
        Liu Shaohui
      3. 11218v2.098.txt
        7 kB


        Liu Shaohui created issue -
        Liu Shaohui made changes -
        Field Original Value New Value
        Assignee Liu Shaohui [ liushaohui ]
        Liu Shaohui made changes -
        Attachment HBASE-11218-trunk-v1.diff [ 12646472 ]
        Liu Shaohui made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jean-Marc Spaggiari made changes -
        Comment [ [~nkeywal]
        I'm curious, why is it an issue for you?
        We use the standalone hbase in our onebox test env. In this env, we start the application and other services (eg, hbase, zookeeper) that it depends on in a machine, , then do many end-to-end logical and failure tests. So it is important to keep data durability in some level. Data loss in standalone mode give a not good impression of HBase to out users.

        On the patch, will this slow down the tests?
        I will do a comparison and show the data later.
        Liu Shaohui made changes -
        Attachment HBASE-11218-trunk-v2.diff [ 12647120 ]
        stack made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Release Note Fix for local filesystem data loss in standalone and pseudo-distributed mode.
        Resolution Fixed [ 1 ]
        stack made changes -
        Attachment 11218v2.098.txt [ 12647176 ]
        Enis Soztutar made changes -
        Status Resolved [ 5 ] Closed [ 6 ]


          • Assignee:
            Liu Shaohui
            Liu Shaohui
          • Votes:
            0 Vote for this issue
            9 Start watching this issue


            • Created: