Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3837

hadop streaming does not use progress reporting to detect hung tasks

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.18.0, 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Changed streaming tasks to adhere to task timeout value specified in the job configuration.

      Description

      Hadoop streaming (StreamJob.java) sets mapred.task.timeout to 0. This effectively means that if tasks hang (either due to bad user code or machine related issue), these tasks never encounters a timeout. This causes the entire job to hang.

      I propose that hadoop streaming not set the timeout to 0. By default, the settings in hadoop-default.xml should be effective for streaming jobs.

        Attachments

        1. streamTimeout.patch
          0.7 kB
          dhruba borthakur
        2. streamTimeout.patch
          0.7 kB
          dhruba borthakur

          Activity

            People

            • Assignee:
              dhruba dhruba borthakur
              Reporter:
              dhruba dhruba borthakur
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: