Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3922

0.22 and 0.23 namenode throws away blocks under construction on restart

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.22.1, 0.23.3
    • Fix Version/s: 0.23.4
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block.

      With append/hsync, this is not acceptable because it may drop sync'ed partial blocks. In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

        Activity

        Hide
        Suresh Srinivas added a comment -

        Kihwal, I just reset Fix Version/s field. It is set when a fix is committed to a branch with the corresponding release number.

        If you intend it for certain releases, add those release numbers to Target Version/s field.

        Show
        Suresh Srinivas added a comment - Kihwal, I just reset Fix Version/s field. It is set when a fix is committed to a branch with the corresponding release number. If you intend it for certain releases, add those release numbers to Target Version/s field.
        Hide
        Kihwal Lee added a comment -

        Posted a patch for branch-0.23.

        Show
        Kihwal Lee added a comment - Posted a patch for branch-0.23.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12545035/hdfs-3922.branch-023.patch.txt
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3184//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545035/hdfs-3922.branch-023.patch.txt against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3184//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        I manually ran test-patch for 0.23. There were actually no additional javadoc warnings introduced by the patch.

        -1 overall.  
        
            +1 @author.  The patch does not contain any @author tags.
        
            +1 tests included.  The patch appears to include 4 new or modified tests.
        
            -1 javadoc.  The javadoc tool appears to have generated 7 warning messages.
        
            +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
        
            +1 eclipse:eclipse.  The patch built with eclipse:eclipse.
        
            +1 findbugs.  The patch does not introduce any new Findbugs (version ) warnings.
        
            +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        
        Show
        Kihwal Lee added a comment - I manually ran test-patch for 0.23. There were actually no additional javadoc warnings introduced by the patch. -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 7 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Daryn Sharp added a comment -

        I've committed to branch-23. Thanks Kihwal!

        I'll let the branch-22 maintainers decide if they want to integrate this change.

        Show
        Daryn Sharp added a comment - I've committed to branch-23. Thanks Kihwal! I'll let the branch-22 maintainers decide if they want to integrate this change.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #388 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/388/)
        HDFS-3922. namenode throws away blocks under construction on restart (Kihwal Lee via daryn) (Revision 1391150)

        Result = UNSTABLE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391150
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #388 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/388/ ) HDFS-3922 . namenode throws away blocks under construction on restart (Kihwal Lee via daryn) (Revision 1391150) Result = UNSTABLE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391150 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development