Hadoop Common
  1. Hadoop Common
  2. HADOOP-3873

DistCp should have an option for limiting the number of files/bytes being copied

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added two new options -filelimit <n> and -sizelimit <n> to DistCp for limiting the total number of files and the total size in bytes, respectively.

      Description

      A single DistCp command may potentially copies a huge number of files/bytes. In such case, DistCp will run a long time and there is no way stop it nicely. It would be good if DistCp have an option to limit the number of files/bytes being copied. Once the limit is reached, DistCp will terminate and return success. All files copied are guaranteed to be good and there is no partially copied file.

      1. 3873_20080808b.patch
        14 kB
        Tsz Wo Nicholas Sze
      2. 3873_20080811b.patch
        31 kB
        Tsz Wo Nicholas Sze
      3. 3873_20080811b_0.18.patch
        31 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Gavin made changes -
          Link This issue is depended upon by HADOOP-3942 [ HADOOP-3942 ]
          Gavin made changes -
          Link This issue blocks HADOOP-3942 [ HADOOP-3942 ]
          Owen O'Malley made changes -
          Component/s tools/distcp [ 12312387 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 3873_20080811b_0.18.patch [ 12397564 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tsz Wo Nicholas Sze made changes -
          Release Note Added two new options -filelimit <n> and -sizelimit <n> to DistCp for limiting the total number of files and the total size in bytes, respectively.
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.19.0 [ 12313211 ]
          Resolution Fixed [ 1 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue blocks HADOOP-3942 [ HADOOP-3942 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HADOOP-3939 [ HADOOP-3939 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 3873_20080811b.patch [ 12388005 ]
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Attachment 3873_20080808b.patch [ 12387862 ]
          Tsz Wo Nicholas Sze created issue -

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development