Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-654

jobs fail with some hardware/system failures on a small number of nodes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.7.2
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      occasionally, such as when the OS is out of some resource, a node fails only partly. The node is up and running, the task tracker is running and sending heartbeats, but every task fails because the tasktracker can't fork tasks or something.
      In these cases, that task tracker keeps getting assigned tasks to execute, and they all fail.
      A couple of nodes like that and jobs start failing badly.

      The job tracker should avoid assigning tasks to tasktrackers that are misbehaving.

      simple approach: avoid tasktrackers that report many more failures than average (say 3X). Simply use the info sent by the TT.
      better but harder: track TT failures over time and:
      1. avoid those that exhibit a high failure rate
      2. tell them to shut down

        Attachments

        1. HADOOP-654_20070221_4.patch
          10 kB
          Arun Murthy
        2. HADOOP-654_20070220_3.patch
          10 kB
          Arun Murthy
        3. HADOOP-654_20070209_2.patch
          10 kB
          Arun Murthy
        4. HADOOP-654_20070208_1.patch
          6 kB
          Arun Murthy

          Activity

            People

            • Assignee:
              acmurthy Arun Murthy
              Reporter:
              yarnon Yoram Arnon
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: