HBase
  1. HBase
  2. HBASE-3696

No durability when running with LocalFileSystem

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: documentation, wal
    • Labels:
      None

      Description

      LocalFileSystem in Hadoop doesn't currently implement sync(), so when we're running in that case, we don't have any durability. This isn't a huge deal since it isn't a realistic deployment scenario, but it's probably worth documenting. It caused some confusion for a user when a table disappeared after killing a standalone instance that was hosting its data in the local FS.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          887d 1h 2m 1 Harsh J 27/Aug/13 03:12
          Patch Available Patch Available Open Open
          260d 5h 20m 1 Harsh J 14/May/14 08:33
          Open Open Resolved Resolved
          13d 20h 35m 1 stack 28/May/14 05:08
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Implemented [ 10 ]
          Hide
          stack added a comment -

          Resolving as implemented by HBASE-11218 (which should go in soon) at least for standalone. Data loss on fs has been doc'd also.

          Hopefully HBASE-11218 will do pseudo distributed mode too. If not, doc will need tweaking. Can do in new issue.

          Show
          stack added a comment - Resolving as implemented by HBASE-11218 (which should go in soon) at least for standalone. Data loss on fs has been doc'd also. Hopefully HBASE-11218 will do pseudo distributed mode too. If not, doc will need tweaking. Can do in new issue.
          Hide
          Misty Stanley-Jones added a comment -

          Can this be closed or can the DOC component be removed, based on the previous comment?

          Show
          Misty Stanley-Jones added a comment - Can this be closed or can the DOC component be removed, based on the previous comment?
          Hide
          Harsh J added a comment -

          Doc patch of mine got applied via HBASE-9379.

          Show
          Harsh J added a comment - Doc patch of mine got applied via HBASE-9379 .
          Harsh J made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Assignee Harsh J [ qwertymaniac ]
          Harsh J made changes -
          Attachment HBASE-3696.patch [ 12600032 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12600032/HBASE-3696.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600032/HBASE-3696.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6917//console This message is automatically generated.
          Harsh J made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Harsh J [ qwertymaniac ]
          Harsh J made changes -
          Attachment HBASE-3696.patch [ 12600032 ]
          Hide
          Harsh J added a comment -

          Is this added foot-note sufficient?

          Show
          Harsh J added a comment - Is this added foot-note sufficient?
          Harsh J made changes -
          Field Original Value New Value
          Link This issue relates to HADOOP-7844 [ HADOOP-7844 ]
          Hide
          Ian Varley added a comment -

          This confused me a bit today while trying to debug HBase replication running locally; you have to roll the HLog in order for ReplicationSource to pick up and replicate your edits. +1 to a fix (but, no idea what that'd be; an implementation of sync() for LocalFileSystem?)

          Show
          Ian Varley added a comment - This confused me a bit today while trying to debug HBase replication running locally; you have to roll the HLog in order for ReplicationSource to pick up and replicate your edits. +1 to a fix (but, no idea what that'd be; an implementation of sync() for LocalFileSystem?)
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #1814 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1814/)
          HBASE-3696 isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #1814 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1814/ ) HBASE-3696 isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase
          Todd Lipcon created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development