Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-302

Maintaining cluster information across multiple job submissions

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Could we have a way to maintain cluster state across multiple job submissions.
      Consider a scenario where we run multiple jobs in iteration on a cluster back to back. The nature of the job is same, but input/output might differ.

      Now, if a node is blacklisted in one iteration of job run, it would be useful to maintain this information and blacklist this node for next iteration of job as well.
      Another situation which we saw is, if there are failures less than mapred.map.max.attempts in each iterations few nodes are never marked for blacklisting. But in we consider two or three iterations, these nodes fail all jobs and should be taken out of cluster. This hampers overall performance of the job.

      Could have have config variables something which matches a job type (provided by user) and maintains the cluster status for that job type alone?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dhruba dhruba borthakur
                Reporter:
                lohit Lohit Vijayarenu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: