Hadoop Common
  1. Hadoop Common
  2. HADOOP-6201

FileSystem::ListStatus should throw FileNotFoundException

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      FileSystem listStatus method throws FileNotFoundException for all implementations. Application code should catch or propagate FileNotFoundException.
      Show
      FileSystem listStatus method throws FileNotFoundException for all implementations. Application code should catch or propagate FileNotFoundException.

      Description

      As discussed in HDFS-538, it would be better for FileSystem and its implementations to throw FileNotFoundException when the file is not found, rather than returning null. This will bring it in line with getFileStatus and, I expect, default user expectations.

        Issue Links

          Activity

          Hide
          Jakob Homan added a comment -

          Made change as discussed in Jira. Surveyed implementaions of FileSystem and found that about half of them were throwing FileNotFound (Har, FTP), the other half were returning null (S3, Local, KFS), so it's good we caught this. There were lots of places null wasn't being checked for and could have eventually resulted in an NPE.

          As far as I know, this is the first patch to cut across all three projects. Uploading patches for HDFS and MR, but they can't be tested without being in a specific order. I have manually tested everything and it all works. Test-patch won't be helpful here.

          In terms of the commiter applying the patches and generating jars:
          1) Apply HADOOP-6201 and generated new common jar (C)
          2) Apply HDFS-538 and pull in C. Won't work because MapReduce jar needs updated. Generate new HDFS jar (H)
          3) Apply MAPREDUCE-895 and pull in C and H. Generate new MapReduce jar (M)
          4) Pull M into HDFS.
          5) Everything should work, but this speaks to the difficulty of generating patches across all three projects.

          Show
          Jakob Homan added a comment - Made change as discussed in Jira. Surveyed implementaions of FileSystem and found that about half of them were throwing FileNotFound (Har, FTP), the other half were returning null (S3, Local, KFS), so it's good we caught this. There were lots of places null wasn't being checked for and could have eventually resulted in an NPE. As far as I know, this is the first patch to cut across all three projects. Uploading patches for HDFS and MR, but they can't be tested without being in a specific order. I have manually tested everything and it all works. Test-patch won't be helpful here. In terms of the commiter applying the patches and generating jars: 1) Apply HADOOP-6201 and generated new common jar (C) 2) Apply HDFS-538 and pull in C. Won't work because MapReduce jar needs updated. Generate new HDFS jar (H) 3) Apply MAPREDUCE-895 and pull in C and H. Generate new MapReduce jar (M) 4) Pull M into HDFS. 5) Everything should work, but this speaks to the difficulty of generating patches across all three projects.
          Hide
          Jakob Homan added a comment -

          submitting patch and praying...

          Show
          Jakob Homan added a comment - submitting patch and praying...
          Hide
          Jakob Homan added a comment -

          Test-patch on common part is clean.

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 11 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec] 
          
          Show
          Jakob Homan added a comment - Test-patch on common part is clean. [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 11 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec]
          Hide
          Chris Douglas added a comment -

          +1

          I committed this. Please add release notes for this and related JIRAs. Thanks Jakob!

          Show
          Chris Douglas added a comment - +1 I committed this. Please add release notes for this and related JIRAs. Thanks Jakob!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/65/)
          . Change FileSystem::listStatus contract to throw
          FileNotFoundException if the directory does not exist, rather than letting
          this be implementation-specific. Contributed by Jakob Homan.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/65/ ) . Change FileSystem::listStatus contract to throw FileNotFoundException if the directory does not exist, rather than letting this be implementation-specific. Contributed by Jakob Homan.
          Hide
          Jakob Homan added a comment -

          Adding release note.

          Show
          Jakob Homan added a comment - Adding release note.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/ )
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development