Description
hadoop cp currently assigns 10 files to copy per map task.
in case of a small number of large files on a large cluster (say 300 files of 30GB each on a 300 node cluster), this results in long execution times.
better would be to assign files per task such that the entire cluster is utilized: one file per map, with a cap of 10000 maps total, so as not to over burden the job tracker.