Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5746

Errors encountered in MROutputThread after the last map/reduce call can go undetected

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      If the child (streaming) process returns successfully and the MROutputThread throws an error, there was no way to detect that as all the IOExceptions was ignored. Such issues can occur when DFS clients were closed etc. Now a check for errors (in threads) is made before finishing off the task and an exception is thrown that fails he task.
      Show
      If the child (streaming) process returns successfully and the MROutputThread throws an error, there was no way to detect that as all the IOExceptions was ignored. Such issues can occur when DFS clients were closed etc. Now a check for errors (in threads) is made before finishing off the task and an exception is thrown that fails he task.

      Description

      The framework map/reduce bridge methods make a check at the beginning of the respective methods whether MROutputThread encountered an exception while writing keys/values that the streaming process emitted. However, if the exception happens in MROutputThread after the last call to the map/reduce method, the exception goes undetected. An example of such an exception is an exception from the DFSClient that fails to write to a file on the HDFS.

        Attachments

        1. 5746.1.patch
          0.7 kB
          Devaraj Das
        2. 5746.6.patch
          0.8 kB
          Amar Kamat
        3. 5746-reproduce.1.patch
          1 kB
          Amar Kamat
        4. 5746-testcase.patch
          3 kB
          Amar Kamat

          Activity

            People

            • Assignee:
              amar_kamat Amar Kamat
              Reporter:
              devaraj Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: