Hadoop Common
  1. Hadoop Common
  2. HADOOP-3837

hadop streaming does not use progress reporting to detect hung tasks

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.18.0, 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Changed streaming tasks to adhere to task timeout value specified in the job configuration.

      Description

      Hadoop streaming (StreamJob.java) sets mapred.task.timeout to 0. This effectively means that if tasks hang (either due to bad user code or machine related issue), these tasks never encounters a timeout. This causes the entire job to hang.

      I propose that hadoop streaming not set the timeout to 0. By default, the settings in hadoop-default.xml should be effective for streaming jobs.

      1. streamTimeout.patch
        0.7 kB
        dhruba borthakur
      2. streamTimeout.patch
        0.7 kB
        dhruba borthakur

        Activity

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            dhruba borthakur
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development