Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2281

NPE in checkpoint during processIOError()

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      At the end of checkpoint BackupNode tries to convergeJournalSpool() and calls revertFileStreams(). The latter closes each file stream, and tries to rename the corresponding file to its permanent location current/edits. If for any reason the rename fails processIOError() is called for failed streams. processIOError() will try to close the stream again and will get NPE in EditLogFileOutputStream.close() because bufCurrent was set to null by the previous close.

      1. BN-bug-NPE.txt
        5 kB
        Konstantin Shvachko
      2. HDFS-2281.1.patch
        6 kB
        Uma Maheswara Rao G
      3. HDFS-2281.patch
        7 kB
        Uma Maheswara Rao G

        Activity

        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Also handled the IOExceptions in SetCheckpointTime in FSImage Storage.
        When i investigate this bug, i found that this issue fixed in trunk already. So, above point back ported from trunk.

        Konstantin, can you just have a look?

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Also handled the IOExceptions in SetCheckpointTime in FSImage Storage. When i investigate this bug, i found that this issue fixed in trunk already. So, above point back ported from trunk. Konstantin, can you just have a look?
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12491816/HDFS-2281.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1162//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491816/HDFS-2281.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1162//console This message is automatically generated.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        This patch is for 0.22 branch. It can not apply on trunk.

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - This patch is for 0.22 branch. It can not apply on trunk.
        Hide
        shv Konstantin Shvachko added a comment -

        Hey Uma. It looks good to me. Could you pls run and post the results of test and test-patch here.

        Show
        shv Konstantin Shvachko added a comment - Hey Uma. It looks good to me. Could you pls run and post the results of test and test-patch here.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        Thanks a lot for taking a look on this patch.
        Below is the test-patch results

        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.

        When i ran the tests no failures observed because of this changes.

        Thanks
        Uma

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Hi Konstantin, Thanks a lot for taking a look on this patch. Below is the test-patch results [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. When i ran the tests no failures observed because of this changes. Thanks Uma
        Hide
        shv Konstantin Shvachko added a comment -

        On the second thought. Why didn't you use processIOError() instead of your new reportErrorsOnDirectories(). Seems to be the appropriate method to call in the case. There is some extra logic in processIOError() related to removing edit streams etc.

        Show
        shv Konstantin Shvachko added a comment - On the second thought. Why didn't you use processIOError() instead of your new reportErrorsOnDirectories(). Seems to be the appropriate method to call in the case. There is some extra logic in processIOError() related to removing edit streams etc.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        Thanks a lot for taking a look.

        Actually this has been re-factored in trunk. So, as part of porting i included them. Actually we can make use of processIOError only.

        So, Used processIOError for handling the failed storage directories.

        Konstantin, can you take a look.

        Test-patch results:

        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.

        ==============================
        Finished
        ==============================

        Thanks
        Uma

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Hi Konstantin, Thanks a lot for taking a look. Actually this has been re-factored in trunk. So, as part of porting i included them. Actually we can make use of processIOError only. So, Used processIOError for handling the failed storage directories. Konstantin, can you take a look. Test-patch results: [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. ============================== Finished ============================== Thanks Uma
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12492780/HDFS-2281.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1190//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492780/HDFS-2281.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1190//console This message is automatically generated.
        Hide
        shv Konstantin Shvachko added a comment -

        +1 I'll commit it to 0.22 branch.

        Show
        shv Konstantin Shvachko added a comment - +1 I'll commit it to 0.22 branch.
        Hide
        shv Konstantin Shvachko added a comment -

        I just committed this. Thank you Uma.

        Show
        shv Konstantin Shvachko added a comment - I just committed this. Thank you Uma.
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #82 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/82/)
        HDFS-2281. NPE in checkpoint during processIOError(). Contributed by Uma Maheswara Rao G.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165943
        Files :

        • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #82 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/82/ ) HDFS-2281 . NPE in checkpoint during processIOError(). Contributed by Uma Maheswara Rao G. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165943 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java

          People

          • Assignee:
            umamaheswararao Uma Maheswara Rao G
            Reporter:
            shv Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development