Hadoop Common
  1. Hadoop Common
  2. HADOOP-3837

hadop streaming does not use progress reporting to detect hung tasks

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.18.0, 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Changed streaming tasks to adhere to task timeout value specified in the job configuration.

      Description

      Hadoop streaming (StreamJob.java) sets mapred.task.timeout to 0. This effectively means that if tasks hang (either due to bad user code or machine related issue), these tasks never encounters a timeout. This causes the entire job to hang.

      I propose that hadoop streaming not set the timeout to 0. By default, the settings in hadoop-default.xml should be effective for streaming jobs.

      1. streamTimeout.patch
        0.7 kB
        dhruba borthakur
      2. streamTimeout.patch
        0.7 kB
        dhruba borthakur

        Activity

        dhruba borthakur created issue -
        dhruba borthakur made changes -
        Field Original Value New Value
        Attachment streamTimeout.patch [ 12387057 ]
        dhruba borthakur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags [Incompatible change]
        Release Note Streaming tasks adhere to task timeout value specified in the job configuration.
        Fix Version/s 0.19.0 [ 12313211 ]
        Fix Version/s 0.18.0 [ 12312972 ]
        dhruba borthakur made changes -
        Attachment streamTimeout.patch [ 12387058 ]
        dhruba borthakur made changes -
        Assignee dhruba borthakur [ dhruba ]
        dhruba borthakur made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Robert Chansler made changes -
        Release Note Streaming tasks adhere to task timeout value specified in the job configuration. Changed streaming tasks to adhere to task timeout value specified in the job configuration.
        Owen O'Malley made changes -
        Component/s contrib/streaming [ 12310972 ]

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            dhruba borthakur
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development