Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3140

JobTracker should not try to promote a (map) task if it does not write to DFS at all

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.17.0
    • None
    • None
    • Reviewed
    • Tasks that don't generate any output are not inserted in the commit queue of the JobTracker. They are marked as SUCCESSFUL by the TaskTracker and the JobTracker updates their state short-circuiting the commit queue.

    Description

      In most cases, map tasks do not write to dfs.
      Thus, when they complete, they should not be put into commit_pending queue at all.
      This will improve the task promotion significantly.

      Attachments

        1. HADOOP-3140-v1.patch
          8 kB
          Amar Kamat
        2. HADOOP-3140-v2.patch
          9 kB
          Amar Kamat
        3. HADOOP-3140-v3.patch
          9 kB
          Amar Kamat
        4. HADOOP-3140-v3.patch
          10 kB
          Amar Kamat

        Activity

          People

            amar_kamat Amar Kamat
            runping Runping Qi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: