Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-895

FileSystem::ListStatus will now throw FileNotFoundException, MapRed needs updated

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The semantics for dealing with non-existent paths passed to FileSystem::listStatus() were updated and solidified in HADOOP-6201 and HDFS-538. Existing code within MapReduce that relied on the previous behavior of some FileSystem implementations of returning null has been updated to catch or propagate a FileNotFoundException, per the method's contract.
      Show
      The semantics for dealing with non-existent paths passed to FileSystem::listStatus() were updated and solidified in HADOOP-6201 and HDFS-538 . Existing code within MapReduce that relied on the previous behavior of some FileSystem implementations of returning null has been updated to catch or propagate a FileNotFoundException, per the method's contract.

      Description

      HADOOP-6201 (and HDFS-538) determined the semantics of FileSystem::ListStatus is not correct and that the actual file system class vary in their implemenations, with some throwing an exception and some returning null. Fixing this will require adjusting code that calls this method.

        Issue Links

          Activity

          Hide
          Jakob Homan added a comment -

          Adding release note.

          Show
          Jakob Homan added a comment - Adding release note.
          Hide
          Chris Douglas added a comment -

          +1

          I committed this. Thanks, Jakob!

          Show
          Chris Douglas added a comment - +1 I committed this. Thanks, Jakob!
          Hide
          Chris Douglas added a comment -

          (after applying HADOOP-6201 , HDFS-538)

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Chris Douglas added a comment - (after applying HADOOP-6201 , HDFS-538 ) [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 9 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Jakob Homan added a comment -

          I should mentioned. I examined every use of the listStatus method in the MR code and updated accordingly. There were several places were we weren't checking for a null and could eventually have it an NPE, but in most cases there's an explicit check if the file exists or not. Not too many changes were required.

          All the unit tests pass except known-bads TestRecoveryManager (MAPREDUCE-880) and TestTrackerLocalization (MAPREDUCE-879). Can't run test-patch as it detects the updated jar files and bails.

          Show
          Jakob Homan added a comment - I should mentioned. I examined every use of the listStatus method in the MR code and updated accordingly. There were several places were we weren't checking for a null and could eventually have it an NPE, but in most cases there's an explicit check if the file exists or not. Not too many changes were required. All the unit tests pass except known-bads TestRecoveryManager ( MAPREDUCE-880 ) and TestTrackerLocalization ( MAPREDUCE-879 ). Can't run test-patch as it detects the updated jar files and bails.
          Hide
          Jakob Homan added a comment -

          Attaching patch, but can't make it patch available because it depends on updated jars from HADOOP-6201 and HDFS-538. Please check notes in HADOOP-6201.

          Show
          Jakob Homan added a comment - Attaching patch, but can't make it patch available because it depends on updated jars from HADOOP-6201 and HDFS-538 . Please check notes in HADOOP-6201 .

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development