Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5872

Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Invalid
    • None
    • None
    • performance
    • Needs to be a Hadoop Jira not a MapReduce Jira

    Description

      In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread. Code is below. This jira will allow a configurable number of threads to be used to issue the copy commands to S3.

      do {
      PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true);
      for (FileMetadata file : listing.getFiles())

      { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length())); }

      priorLastKey = listing.getPriorLastKey();
      } while (priorLastKey != null);

      Attachments

        Activity

          People

            ted.m Theodore michael Malaska
            ted.m Theodore michael Malaska
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: