Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3150

Move task file promotion into the task

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.19.0
    • None
    • None
    • Incompatible change
    • Hide
      Moved task file promotion to the Task. When the task has finished, it will do a commit and is declared SUCCEDED. Job cleanup is done by a separate task. Job is declared SUCCEDED/FAILED after the cleanup task has finished. Added public classes org.apache.hadoop.mapred.JobContext, TaskAttemptContext, OutputCommitter and FileOutputCommiitter. Added public APIs: public OutputCommitter getOutputCommitter() and
      public void setOutputCommitter(Class<? extends OutputCommitter> theClass) in org.apache.hadoop.mapred.JobConf
      Show
      Moved task file promotion to the Task. When the task has finished, it will do a commit and is declared SUCCEDED. Job cleanup is done by a separate task. Job is declared SUCCEDED/FAILED after the cleanup task has finished. Added public classes org.apache.hadoop.mapred.JobContext, TaskAttemptContext, OutputCommitter and FileOutputCommiitter. Added public APIs: public OutputCommitter getOutputCommitter() and public void setOutputCommitter(Class<? extends OutputCommitter> theClass) in org.apache.hadoop.mapred.JobConf

    Description

      We need to move the task file promotion from the JobTracker to the Task and move it down into the output format.

      Attachments

        1. 3150.patch
          75 kB
          Devaraj Das
        2. 3150 test plan.txt
          2 kB
          Amareshwari Sriramadasu
        3. patch-3150.txt
          126 kB
          Amareshwari Sriramadasu
        4. patch-3150.txt
          124 kB
          Amareshwari Sriramadasu
        5. patch-3150.txt
          124 kB
          Amareshwari Sriramadasu
        6. patch-3150.txt
          120 kB
          Amareshwari Sriramadasu
        7. patch-3150.txt
          113 kB
          Amareshwari Sriramadasu
        8. patch-3150.txt
          109 kB
          Amareshwari Sriramadasu
        9. patch-3150.txt
          110 kB
          Amareshwari Sriramadasu
        10. patch-3150.txt
          106 kB
          Amareshwari Sriramadasu
        11. patch-3150.txt
          105 kB
          Amareshwari Sriramadasu
        12. patch-3150.txt
          89 kB
          Amareshwari Sriramadasu
        13. patch-3150.txt
          85 kB
          Amareshwari Sriramadasu
        14. patch-3150.txt
          85 kB
          Amareshwari Sriramadasu
        15. patch-3150.txt
          88 kB
          Amareshwari Sriramadasu
        16. patch-3150.txt
          82 kB
          Amareshwari Sriramadasu
        17. patch-3150.txt
          82 kB
          Amareshwari Sriramadasu
        18. patch-3150.txt
          77 kB
          Amareshwari Sriramadasu
        19. patch-3150.txt
          77 kB
          Amareshwari Sriramadasu
        20. patch-3150.txt
          73 kB
          Amareshwari Sriramadasu
        21. patch-3150.txt
          116 kB
          Amareshwari Sriramadasu
        22. patch-3150.txt
          116 kB
          Amareshwari Sriramadasu
        23. patch-3150.txt
          116 kB
          Amareshwari Sriramadasu
        24. patch-3150.txt
          110 kB
          Amareshwari Sriramadasu
        25. patch-3150.txt
          105 kB
          Amareshwari Sriramadasu

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-3149 supporting multiple outputs for M/R jobs

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4100 Scheduler.assignTasks should not be dealing with cleanupTask

          • Major - Major loss of function.
          • Closed

          Activity

            People

              amareshwari Amareshwari Sriramadasu
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: