Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2281

NPE in checkpoint during processIOError()

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      At the end of checkpoint BackupNode tries to convergeJournalSpool() and calls revertFileStreams(). The latter closes each file stream, and tries to rename the corresponding file to its permanent location current/edits. If for any reason the rename fails processIOError() is called for failed streams. processIOError() will try to close the stream again and will get NPE in EditLogFileOutputStream.close() because bufCurrent was set to null by the previous close.

      1. BN-bug-NPE.txt
        5 kB
        Konstantin Shvachko
      2. HDFS-2281.1.patch
        6 kB
        Uma Maheswara Rao G
      3. HDFS-2281.patch
        7 kB
        Uma Maheswara Rao G

        Activity

        Hide
        Uma Maheswara Rao G added a comment -

        Also handled the IOExceptions in SetCheckpointTime in FSImage Storage.
        When i investigate this bug, i found that this issue fixed in trunk already. So, above point back ported from trunk.

        Konstantin, can you just have a look?

        Show
        Uma Maheswara Rao G added a comment - Also handled the IOExceptions in SetCheckpointTime in FSImage Storage. When i investigate this bug, i found that this issue fixed in trunk already. So, above point back ported from trunk. Konstantin, can you just have a look?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12491816/HDFS-2281.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1162//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491816/HDFS-2281.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1162//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        This patch is for 0.22 branch. It can not apply on trunk.

        Show
        Uma Maheswara Rao G added a comment - This patch is for 0.22 branch. It can not apply on trunk.
        Hide
        Konstantin Shvachko added a comment -

        Hey Uma. It looks good to me. Could you pls run and post the results of test and test-patch here.

        Show
        Konstantin Shvachko added a comment - Hey Uma. It looks good to me. Could you pls run and post the results of test and test-patch here.
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        Thanks a lot for taking a look on this patch.
        Below is the test-patch results

        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.

        When i ran the tests no failures observed because of this changes.

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Konstantin, Thanks a lot for taking a look on this patch. Below is the test-patch results [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. When i ran the tests no failures observed because of this changes. Thanks Uma
        Hide
        Konstantin Shvachko added a comment -

        On the second thought. Why didn't you use processIOError() instead of your new reportErrorsOnDirectories(). Seems to be the appropriate method to call in the case. There is some extra logic in processIOError() related to removing edit streams etc.

        Show
        Konstantin Shvachko added a comment - On the second thought. Why didn't you use processIOError() instead of your new reportErrorsOnDirectories(). Seems to be the appropriate method to call in the case. There is some extra logic in processIOError() related to removing edit streams etc.
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        Thanks a lot for taking a look.

        Actually this has been re-factored in trunk. So, as part of porting i included them. Actually we can make use of processIOError only.

        So, Used processIOError for handling the failed storage directories.

        Konstantin, can you take a look.

        Test-patch results:

        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.

        ==============================
        Finished
        ==============================

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Konstantin, Thanks a lot for taking a look. Actually this has been re-factored in trunk. So, as part of porting i included them. Actually we can make use of processIOError only. So, Used processIOError for handling the failed storage directories. Konstantin, can you take a look. Test-patch results: [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. ============================== Finished ============================== Thanks Uma
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12492780/HDFS-2281.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1190//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492780/HDFS-2281.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1190//console This message is automatically generated.
        Hide
        Konstantin Shvachko added a comment -

        +1 I'll commit it to 0.22 branch.

        Show
        Konstantin Shvachko added a comment - +1 I'll commit it to 0.22 branch.
        Hide
        Konstantin Shvachko added a comment -

        I just committed this. Thank you Uma.

        Show
        Konstantin Shvachko added a comment - I just committed this. Thank you Uma.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #82 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/82/)
        HDFS-2281. NPE in checkpoint during processIOError(). Contributed by Uma Maheswara Rao G.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165943
        Files :

        • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #82 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/82/ ) HDFS-2281 . NPE in checkpoint during processIOError(). Contributed by Uma Maheswara Rao G. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165943 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development