Hadoop Common
  1. Hadoop Common
  2. HADOOP-3361

Implement renames for NativeS3FileSystem

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Amazon S3 now supports a copy object operation (http://docs.amazonwebservices.com/AmazonS3/2006-03-01/UsingCopyingObjects.html). We can use this to properly support renames in NativeS3FileSystem.

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          The Jets3t library (http://jets3t.s3.amazonaws.com/index.html) has added support for copy (and rename), but not in a released version yet.

          Show
          Tom White added a comment - The Jets3t library ( http://jets3t.s3.amazonaws.com/index.html ) has added support for copy (and rename), but not in a released version yet.
          Hide
          Tom White added a comment -

          Here's a patch based on Albert's work in HADOOP-3912, with a couple of modifications to make the tests pass. This should not be committed until there's a new jets3t release.

          This change is needed to use s3n as a MapReduce destination.

          Show
          Tom White added a comment - Here's a patch based on Albert's work in HADOOP-3912 , with a couple of modifications to make the tests pass. This should not be committed until there's a new jets3t release. This change is needed to use s3n as a MapReduce destination.
          Hide
          Tom White added a comment -

          Resubmitting to hudson.

          Show
          Tom White added a comment - Resubmitting to hudson.
          Hide
          Tom White added a comment -

          Trying hudson again.

          Show
          Tom White added a comment - Trying hudson again.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12387820/hadoop-3361.patch
          against trunk revision 689548.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3124/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3124/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3124/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3124/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12387820/hadoop-3361.patch against trunk revision 689548. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3124/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3124/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3124/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3124/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          Tom,
          Can you resubmit this patch when the new version is released? Please attach the jar to the jira when it is available.

          Show
          Owen O'Malley added a comment - Tom, Can you resubmit this patch when the new version is released? Please attach the jar to the jira when it is available.
          Hide
          Tom White added a comment -

          Owen,
          Sorry, I wasn't clear. The new version of JetS3t (0.6.1) with rename support has been released and I committed it before submitting the patch in this issue to Hudon. The test failures are unrelated, so I believe this can be committed. I'll do it in the next couple of days unless anyone objects.

          Show
          Tom White added a comment - Owen, Sorry, I wasn't clear. The new version of JetS3t (0.6.1) with rename support has been released and I committed it before submitting the patch in this issue to Hudon. The test failures are unrelated, so I believe this can be committed. I'll do it in the next couple of days unless anyone objects.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Albert!

          Show
          Tom White added a comment - I've just committed this. Thanks Albert!
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #597 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/597/ )

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development