Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-229

hadoop cp should generate a better number of map tasks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 0.3.0
    • fs
    • None

    Description

      hadoop cp currently assigns 10 files to copy per map task.
      in case of a small number of large files on a large cluster (say 300 files of 30GB each on a 300 node cluster), this results in long execution times.
      better would be to assign files per task such that the entire cluster is utilized: one file per map, with a cap of 10000 maps total, so as not to over burden the job tracker.

      Attachments

        Activity

          People

            milindb Milind Barve
            yarnon Yoram Arnon
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: