Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-654

jobs fail with some hardware/system failures on a small number of nodes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.7.2
    • 0.12.0
    • None
    • None

    Description

      occasionally, such as when the OS is out of some resource, a node fails only partly. The node is up and running, the task tracker is running and sending heartbeats, but every task fails because the tasktracker can't fork tasks or something.
      In these cases, that task tracker keeps getting assigned tasks to execute, and they all fail.
      A couple of nodes like that and jobs start failing badly.

      The job tracker should avoid assigning tasks to tasktrackers that are misbehaving.

      simple approach: avoid tasktrackers that report many more failures than average (say 3X). Simply use the info sent by the TT.
      better but harder: track TT failures over time and:
      1. avoid those that exhibit a high failure rate
      2. tell them to shut down

      Attachments

        1. HADOOP-654_20070208_1.patch
          6 kB
          Arun Murthy
        2. HADOOP-654_20070209_2.patch
          10 kB
          Arun Murthy
        3. HADOOP-654_20070220_3.patch
          10 kB
          Arun Murthy
        4. HADOOP-654_20070221_4.patch
          10 kB
          Arun Murthy

        Activity

          People

            acmurthy Arun Murthy
            yarnon Yoram Arnon
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: