Hadoop Common
  1. Hadoop Common
  2. HADOOP-10560

Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs/s3
    • Target Version/s:

      Description

      In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread. Code is below. This jira will allow a configurable number of threads to be used to issue the copy commands to S3.
      do {
      PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true);
      for (FileMetadata file : listing.getFiles())

      { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length())); }

      priorLastKey = listing.getPriorLastKey();
      } while (priorLastKey != null);

      1. HADOOP-10560-1.patch
        6 kB
        Ted Malaska
      2. HADOOP-10560.patch
        3 kB
        Ted Malaska

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Ted Malaska
            Reporter:
            Ted Malaska
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development