Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10560

Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs/s3
    • Target Version/s:

      Description

      In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread. Code is below. This jira will allow a configurable number of threads to be used to issue the copy commands to S3.
      do

      { PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true); for (FileMetadata file : listing.getFiles()) { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length())); }

      priorLastKey = listing.getPriorLastKey();
      } while (priorLastKey != null);

        Attachments

        1. HADOOP-10560-1.patch
          6 kB
          Theodore michael Malaska
        2. HADOOP-10560.patch
          3 kB
          Theodore michael Malaska

          Activity

            People

            • Assignee:
              ted.m Theodore michael Malaska
              Reporter:
              ted.m Theodore michael Malaska
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: