Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-895

FileSystem::ListStatus will now throw FileNotFoundException, MapRed needs updated

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The semantics for dealing with non-existent paths passed to FileSystem::listStatus() were updated and solidified in HADOOP-6201 and HDFS-538. Existing code within MapReduce that relied on the previous behavior of some FileSystem implementations of returning null has been updated to catch or propagate a FileNotFoundException, per the method's contract.
      Show
      The semantics for dealing with non-existent paths passed to FileSystem::listStatus() were updated and solidified in HADOOP-6201 and HDFS-538 . Existing code within MapReduce that relied on the previous behavior of some FileSystem implementations of returning null has been updated to catch or propagate a FileNotFoundException, per the method's contract.

      Description

      HADOOP-6201 (and HDFS-538) determined the semantics of FileSystem::ListStatus is not correct and that the actual file system class vary in their implemenations, with some throwing an exception and some returning null. Fixing this will require adjusting code that calls this method.

        Issue Links

          Activity

          Jakob Homan created issue -
          Jakob Homan made changes -
          Field Original Value New Value
          Link This issue depends on HADOOP-6201 [ HADOOP-6201 ]
          Jakob Homan made changes -
          Link This issue depends upon HDFS-538 [ HDFS-538 ]
          Hide
          Jakob Homan added a comment -

          Attaching patch, but can't make it patch available because it depends on updated jars from HADOOP-6201 and HDFS-538. Please check notes in HADOOP-6201.

          Show
          Jakob Homan added a comment - Attaching patch, but can't make it patch available because it depends on updated jars from HADOOP-6201 and HDFS-538 . Please check notes in HADOOP-6201 .
          Jakob Homan made changes -
          Attachment MAPREDUCE-895.patch [ 12417305 ]
          Hide
          Jakob Homan added a comment -

          I should mentioned. I examined every use of the listStatus method in the MR code and updated accordingly. There were several places were we weren't checking for a null and could eventually have it an NPE, but in most cases there's an explicit check if the file exists or not. Not too many changes were required.

          All the unit tests pass except known-bads TestRecoveryManager (MAPREDUCE-880) and TestTrackerLocalization (MAPREDUCE-879). Can't run test-patch as it detects the updated jar files and bails.

          Show
          Jakob Homan added a comment - I should mentioned. I examined every use of the listStatus method in the MR code and updated accordingly. There were several places were we weren't checking for a null and could eventually have it an NPE, but in most cases there's an explicit check if the file exists or not. Not too many changes were required. All the unit tests pass except known-bads TestRecoveryManager ( MAPREDUCE-880 ) and TestTrackerLocalization ( MAPREDUCE-879 ). Can't run test-patch as it detects the updated jar files and bails.
          Hide
          Chris Douglas added a comment -

          (after applying HADOOP-6201 , HDFS-538)

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Chris Douglas added a comment - (after applying HADOOP-6201 , HDFS-538 ) [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 9 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Chris Douglas added a comment -

          +1

          I committed this. Thanks, Jakob!

          Show
          Chris Douglas added a comment - +1 I committed this. Thanks, Jakob!
          Chris Douglas made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Fix Version/s 0.21.0 [ 12314045 ]
          Resolution Fixed [ 1 ]
          Hide
          Jakob Homan added a comment -

          Adding release note.

          Show
          Jakob Homan added a comment - Adding release note.
          Jakob Homan made changes -
          Release Note The semantics for dealing with non-existent paths passed to FileSystem::listStatus() were updated and solidified in HADOOP-6201 and HDFS-538. Existing code within MapReduce that relied on the previous behavior of some FileSystem implementations of returning null has been updated to catch or propagate a FileNotFoundException, per the method's contract.
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue depends on HADOOP-6201 [ HADOOP-6201 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-6201 [ HADOOP-6201 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          21h 50m 1 Chris Douglas 21/Aug/09 23:51
          Resolved Resolved Closed Closed
          367d 22h 24m 1 Tom White 24/Aug/10 22:15

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development