Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2281

NPE in checkpoint during processIOError()

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      At the end of checkpoint BackupNode tries to convergeJournalSpool() and calls revertFileStreams(). The latter closes each file stream, and tries to rename the corresponding file to its permanent location current/edits. If for any reason the rename fails processIOError() is called for failed streams. processIOError() will try to close the stream again and will get NPE in EditLogFileOutputStream.close() because bufCurrent was set to null by the previous close.

      1. BN-bug-NPE.txt
        5 kB
        Konstantin Shvachko
      2. HDFS-2281.1.patch
        6 kB
        Uma Maheswara Rao G
      3. HDFS-2281.patch
        7 kB
        Uma Maheswara Rao G

        Activity

        Konstantin Shvachko created issue -
        Konstantin Shvachko made changes -
        Field Original Value New Value
        Attachment BN-bug-NPE.txt [ 12491429 ]
        Uma Maheswara Rao G made changes -
        Assignee Uma Maheswara Rao G [ umamaheswararao ]
        Uma Maheswara Rao G made changes -
        Attachment HDFS-2281.patch [ 12491816 ]
        Hide
        Uma Maheswara Rao G added a comment -

        Also handled the IOExceptions in SetCheckpointTime in FSImage Storage.
        When i investigate this bug, i found that this issue fixed in trunk already. So, above point back ported from trunk.

        Konstantin, can you just have a look?

        Show
        Uma Maheswara Rao G added a comment - Also handled the IOExceptions in SetCheckpointTime in FSImage Storage. When i investigate this bug, i found that this issue fixed in trunk already. So, above point back ported from trunk. Konstantin, can you just have a look?
        Uma Maheswara Rao G made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12491816/HDFS-2281.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1162//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491816/HDFS-2281.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1162//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        This patch is for 0.22 branch. It can not apply on trunk.

        Show
        Uma Maheswara Rao G added a comment - This patch is for 0.22 branch. It can not apply on trunk.
        Hide
        Konstantin Shvachko added a comment -

        Hey Uma. It looks good to me. Could you pls run and post the results of test and test-patch here.

        Show
        Konstantin Shvachko added a comment - Hey Uma. It looks good to me. Could you pls run and post the results of test and test-patch here.
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        Thanks a lot for taking a look on this patch.
        Below is the test-patch results

        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.

        When i ran the tests no failures observed because of this changes.

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Konstantin, Thanks a lot for taking a look on this patch. Below is the test-patch results [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. When i ran the tests no failures observed because of this changes. Thanks Uma
        Uma Maheswara Rao G made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Uma Maheswara Rao G made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note Hi Konstantin,
        Can you pls take a look on this?
        Hide
        Konstantin Shvachko added a comment -

        On the second thought. Why didn't you use processIOError() instead of your new reportErrorsOnDirectories(). Seems to be the appropriate method to call in the case. There is some extra logic in processIOError() related to removing edit streams etc.

        Show
        Konstantin Shvachko added a comment - On the second thought. Why didn't you use processIOError() instead of your new reportErrorsOnDirectories(). Seems to be the appropriate method to call in the case. There is some extra logic in processIOError() related to removing edit streams etc.
        Uma Maheswara Rao G made changes -
        Release Note Hi Konstantin,
        Can you pls take a look on this?
        Uma Maheswara Rao G made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Uma Maheswara Rao G made changes -
        Attachment HDFS-2281.1.patch [ 12492780 ]
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        Thanks a lot for taking a look.

        Actually this has been re-factored in trunk. So, as part of porting i included them. Actually we can make use of processIOError only.

        So, Used processIOError for handling the failed storage directories.

        Konstantin, can you take a look.

        Test-patch results:

        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.

        ==============================
        Finished
        ==============================

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Konstantin, Thanks a lot for taking a look. Actually this has been re-factored in trunk. So, as part of porting i included them. Actually we can make use of processIOError only. So, Used processIOError for handling the failed storage directories. Konstantin, can you take a look. Test-patch results: [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. ============================== Finished ============================== Thanks Uma
        Uma Maheswara Rao G made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12492780/HDFS-2281.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1190//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492780/HDFS-2281.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1190//console This message is automatically generated.
        Hide
        Konstantin Shvachko added a comment -

        +1 I'll commit it to 0.22 branch.

        Show
        Konstantin Shvachko added a comment - +1 I'll commit it to 0.22 branch.
        Hide
        Konstantin Shvachko added a comment -

        I just committed this. Thank you Uma.

        Show
        Konstantin Shvachko added a comment - I just committed this. Thank you Uma.
        Konstantin Shvachko made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #82 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/82/)
        HDFS-2281. NPE in checkpoint during processIOError(). Contributed by Uma Maheswara Rao G.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165943
        Files :

        • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #82 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/82/ ) HDFS-2281 . NPE in checkpoint during processIOError(). Contributed by Uma Maheswara Rao G. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165943 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        7d 5m 2 Uma Maheswara Rao G 02/Sep/11 17:56
        Open Open Patch Available Patch Available
        2d 16h 38m 3 Uma Maheswara Rao G 02/Sep/11 18:03
        Patch Available Patch Available Resolved Resolved
        4d 6h 33m 1 Konstantin Shvachko 07/Sep/11 00:36

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development