Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-43 Ignoring IOExceptions on close
  3. HDFS-2330

In NNStorage.java, IOExceptions of stream closures can mask root exceptions.

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.24.0
    • Fix Version/s: 3.0.0, 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In NNStorage.java:
      There are many stream closures in finally block.
      There is a chance that they can mask the root exceptions.

      So, better to follow the pattern like below:

        try{
           ............
           ............
           stream.close();
           stream =null;
         }
         finally{
           IOUtils.cleanup(LOG, stream);
         }
      
      1. HDFS-2330.patch
        2 kB
        Uma Maheswara Rao G
      2. HDFS-2330.patch
        2 kB
        Uma Maheswara Rao G

        Activity

        Hide
        Uma Maheswara Rao G added a comment -

        Updated a patch for review!

        Show
        Uma Maheswara Rao G added a comment - Updated a patch for review!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494372/HDFS-2330.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.TestDfsOverAvroRpc
        org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
        org.apache.hadoop.hdfs.server.datanode.TestReplicasMap

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1239//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1239//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1239//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494372/HDFS-2330.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap org.apache.hadoop.hdfs.server.datanode.TestReplicasMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1239//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1239//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1239//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        Test failures are not related. Since the changes are related to stream closures, not provided any special tests for this change.

        Show
        Uma Maheswara Rao G added a comment - Test failures are not related. Since the changes are related to stream closures, not provided any special tests for this change.
        Hide
        Uma Maheswara Rao G added a comment -

        Moved the closes to before return.

        +      oldFile.close();
        +      oldFile = null;
               if (oldVersion < LAST_PRE_UPGRADE_LAYOUT_VERSION)
                 return false;
        
        Show
        Uma Maheswara Rao G added a comment - Moved the closes to before return. + oldFile.close(); + oldFile = null ; if (oldVersion < LAST_PRE_UPGRADE_LAYOUT_VERSION) return false ;
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494467/HDFS-2330.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.TestDfsOverAvroRpc
        org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1250//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1250//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1250//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494467/HDFS-2330.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1250//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1250//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1250//console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Uma!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Uma!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #987 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/987/)
        HDFS-2330. In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #987 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/987/ ) HDFS-2330 . In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #927 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/927/)
        HDFS-2330. In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #927 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/927/ ) HDFS-2330 . In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #910 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/910/)
        HDFS-2330. In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #910 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/910/ ) HDFS-2330 . In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #804 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/804/)
        HDFS-2330. In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #804 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/804/ ) HDFS-2330 . In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #834 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/834/)
        HDFS-2330. In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #834 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/834/ ) HDFS-2330 . In NNStorage and FSImagePreTransactionalStorageInspector, IOExceptions of stream closures can mask root exceptions. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172219 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Hide
        Todd Lipcon added a comment -

        Backported this small fix to branch-2 to avoid some merge conflicts in further backports.

        Show
        Todd Lipcon added a comment - Backported this small fix to branch-2 to avoid some merge conflicts in further backports.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2639 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2639/)
        Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2639 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2639/ ) Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2574 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2574/)
        Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2574 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2574/ ) Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2596 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2596/)
        Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2596 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2596/ ) Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/)
        Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/ ) Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/)
        Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/ ) Move HDFS-2330 and HDFS-3190 to branch-2 section, since they have been backported from trunk. (Revision 1372605) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372605 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Uma Maheswara Rao G
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development