Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1934

Fix NullPointerException when File.listFiles() API returns null

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None

      Description

      While testing Disk Fail Inplace, We encountered the NPE from this part of the code.

      File[] files = dir.listFiles();
      for (File f : files)

      { ... }

      This is kinda of an API issue. When a disk is bad (or name is not a directory), this API (listFiles, list) return null rather than throwing an exception. This 'for loop' throws a NPE exception. And same applies to dir.list() API.

      Fix all the places where null condition was not checked.

      1. HDFS-1934-5.patch
        4 kB
        Bharath Mundlapudi
      2. HDFS-1934-4.patch
        4 kB
        Bharath Mundlapudi
      3. HDFS-1934-3.patch
        4 kB
        Bharath Mundlapudi
      4. HDFS-1934-2.patch
        4 kB
        Bharath Mundlapudi
      5. HDFS-1934-1.patch
        2 kB
        Bharath Mundlapudi

        Issue Links

          Activity

          Hide
          Bharath Mundlapudi added a comment -

          Attaching a patch which addresses this problem.

          Show
          Bharath Mundlapudi added a comment - Attaching a patch which addresses this problem.
          Hide
          Bharath Mundlapudi added a comment -

          Adding this check at another location. So the updated patch.

          Show
          Bharath Mundlapudi added a comment - Adding this check at another location. So the updated patch.
          Hide
          Jakob Homan added a comment -

          This semantic was argued out in HDFS-538, where it was agreed that in general empty collections or arrays should be returned for directories/whatever with no contents, exceptions should be thrown for invalid/non-existent requests and null should never be returned. Glancing at the latest patch, I see some hard tabs and some odd spacing.

          Show
          Jakob Homan added a comment - This semantic was argued out in HDFS-538 , where it was agreed that in general empty collections or arrays should be returned for directories/whatever with no contents, exceptions should be thrown for invalid/non-existent requests and null should never be returned. Glancing at the latest patch, I see some hard tabs and some odd spacing.
          Hide
          Bharath Mundlapudi added a comment -

          Thanks for reviewing this patch, Jakob.

          Show
          Bharath Mundlapudi added a comment - Thanks for reviewing this patch, Jakob.
          Hide
          Bharath Mundlapudi added a comment -

          Right, this patch is trying to address exactly what you have mentioned.

          Show
          Bharath Mundlapudi added a comment - Right, this patch is trying to address exactly what you have mentioned.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480631/HDFS-1934-3.patch
          against trunk revision 1128009.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/642//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/642//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/642//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480631/HDFS-1934-3.patch against trunk revision 1128009. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/642//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/642//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/642//console This message is automatically generated.
          Hide
          Matt Foley added a comment -

          Minor: This is an old idiom: LOG.warn("Exception occured while compiling report: " + StringUtils.stringifyException(ioe));
          The preferred form is: LOG.warn("Exception occurred while compiling report: ", ioe);

          All the changes you made look like good catches.
          Make that edit and I'll commit.

          Show
          Matt Foley added a comment - Minor: This is an old idiom: LOG.warn("Exception occured while compiling report: " + StringUtils.stringifyException(ioe)); The preferred form is: LOG.warn("Exception occurred while compiling report: ", ioe); All the changes you made look like good catches. Make that edit and I'll commit.
          Hide
          Bharath Mundlapudi added a comment -

          Thanks for reviewing, Matt. Attaching the patch with this change.

          Show
          Bharath Mundlapudi added a comment - Thanks for reviewing, Matt. Attaching the patch with this change.
          Hide
          Bharath Mundlapudi added a comment -

          Reattaching a patch with minor correction for logging.

          Show
          Bharath Mundlapudi added a comment - Reattaching a patch with minor correction for logging.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481005/HDFS-1934-5.patch
          against trunk revision 1129942.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.TestDFSUpgradeFromImage

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/669//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/669//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/669//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481005/HDFS-1934-5.patch against trunk revision 1129942. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.TestDFSUpgradeFromImage +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/669//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/669//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/669//console This message is automatically generated.
          Hide
          Matt Foley added a comment -

          The test failures are unrelated.

          +1. Committed to trunk. Thanks Bharath! And thanks to Jakob for reviewing.

          Show
          Matt Foley added a comment - The test failures are unrelated. +1. Committed to trunk. Thanks Bharath! And thanks to Jakob for reviewing.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #685 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/)
          HDFS-1934. Fix NullPointerException when certain File APIs return null. Contributed by Bharath Mundlapudi.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130262
          Files :

          • /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          • /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
          • /hadoop/hdfs/trunk/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #685 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/ ) HDFS-1934 . Fix NullPointerException when certain File APIs return null. Contributed by Bharath Mundlapudi. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130262 Files : /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/hdfs/trunk/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #704 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #704 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/ )

            People

            • Assignee:
              Bharath Mundlapudi
              Reporter:
              Bharath Mundlapudi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development