Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5872

Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: performance
    • Labels:
    • Release Note:
      Needs to be a Hadoop Jira not a MapReduce Jira

      Description

      In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread. Code is below. This jira will allow a configurable number of threads to be used to issue the copy commands to S3.

      do {
      PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true);
      for (FileMetadata file : listing.getFiles())

      { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length())); }

      priorLastKey = listing.getPriorLastKey();
      } while (priorLastKey != null);

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Ted Malaska
            Reporter:
            Ted Malaska
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development