Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3837

hadop streaming does not use progress reporting to detect hung tasks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.17.0
    • 0.18.0, 0.19.0
    • None
    • None
    • Incompatible change
    • Changed streaming tasks to adhere to task timeout value specified in the job configuration.

    Description

      Hadoop streaming (StreamJob.java) sets mapred.task.timeout to 0. This effectively means that if tasks hang (either due to bad user code or machine related issue), these tasks never encounters a timeout. This causes the entire job to hang.

      I propose that hadoop streaming not set the timeout to 0. By default, the settings in hadoop-default.xml should be effective for streaming jobs.

      Attachments

        1. streamTimeout.patch
          0.7 kB
          Dhruba Borthakur
        2. streamTimeout.patch
          0.7 kB
          Dhruba Borthakur

        Activity

          People

            dhruba Dhruba Borthakur
            dhruba Dhruba Borthakur
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: