Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-491

streaming jobs should allow programs that don't do any IO for a long time

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      The jobtracker relies on task to send heartbeats to know the tasks are still alive.
      There is a 600 seconds timeout preset.
      hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
      Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.

      It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.

      There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.

        Attachments

        1. HADOOP-491_20070212_3.patch
          4 kB
          Arun C Murthy
        2. HADOOP-491_20070206_2.patch
          3 kB
          Arun C Murthy
        3. HADOOP-491_20070205_1.patch
          4 kB
          Arun C Murthy

          Activity

            People

            • Assignee:
              acmurthy Arun C Murthy
              Reporter:
              arkady arkady borkovsky
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: