Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HIVE-15199 shows that s3a allows rename onto an existing file, which is something HDFS, azure and s3n do not permit (though file:// does). This is breaking bits of Hive, is an inconsistency with HDFS and a regression compared to s3n semantics.

      I propose: rejecting the rename on a file -> file rename if the destination exists (easy) and changing the s3a.xml contract file to declare the behavior change; this is needed for AbstractContractRenameTest.testRenameFileOverExistingFile to handle the changed semantics.

      1. HADOOP-13813-branch-2-001.patch
        10 kB
        Steve Loughran
      2. HADOOP-13823-branch-2-002.patch
        10 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Also: s3a doesn't raise an exception if the source isn't there. That's at odds with the FS Specification

          Show
          stevel@apache.org Steve Loughran added a comment - Also: s3a doesn't raise an exception if the source isn't there. That's at odds with the FS Specification
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 001. Modified behaviour; XML declaration and removed subclass of contract test

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 001. Modified behaviour; XML declaration and removed subclass of contract test
          Hide
          stevel@apache.org Steve Loughran added a comment -

          testing: s3a ireland

          Show
          stevel@apache.org Steve Loughran added a comment - testing: s3a ireland
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 7s branch-2 passed
          +1 compile 0m 20s branch-2 passed with JDK v1.8.0_111
          +1 compile 0m 19s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 14s branch-2 passed
          +1 mvnsite 0m 25s branch-2 passed
          +1 mvneclipse 0m 21s branch-2 passed
          +1 findbugs 0m 37s branch-2 passed
          +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_111
          +1 javadoc 0m 16s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 20s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_111
          +1 javac 0m 17s the patch passed
          +1 compile 0m 18s the patch passed with JDK v1.7.0_121
          +1 javac 0m 18s the patch passed
          -0 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 0m 44s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_111
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_121
          +1 unit 0m 21s hadoop-aws in the patch passed with JDK v1.7.0_121.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          15m 54s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13823
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840436/HADOOP-13813-branch-2-001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 3497edcfbf3d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / c28dc5e
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11133/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11133/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11133/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 7s branch-2 passed +1 compile 0m 20s branch-2 passed with JDK v1.8.0_111 +1 compile 0m 19s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 14s branch-2 passed +1 mvnsite 0m 25s branch-2 passed +1 mvneclipse 0m 21s branch-2 passed +1 findbugs 0m 37s branch-2 passed +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_111 +1 javadoc 0m 16s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 20s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_111 +1 javac 0m 17s the patch passed +1 compile 0m 18s the patch passed with JDK v1.7.0_121 +1 javac 0m 18s the patch passed -0 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_111 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_121 +1 unit 0m 21s hadoop-aws in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 15m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13823 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840436/HADOOP-13813-branch-2-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 3497edcfbf3d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / c28dc5e Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11133/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11133/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11133/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          checkstyle is being fussy about param names, but may as well fix that

          Show
          stevel@apache.org Steve Loughran added a comment - checkstyle is being fussy about param names, but may as well fix that
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 2; patch 1 with the parameter to a method with a changed name.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 2; patch 1 with the parameter to a method with a changed name.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          rested: s3 ireland

          Show
          stevel@apache.org Steve Loughran added a comment - rested: s3 ireland
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 37s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 9s branch-2 passed
          +1 compile 0m 18s branch-2 passed with JDK v1.8.0_111
          +1 compile 0m 18s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 13s branch-2 passed
          +1 mvnsite 0m 23s branch-2 passed
          +1 mvneclipse 0m 20s branch-2 passed
          +1 findbugs 0m 34s branch-2 passed
          +1 javadoc 0m 13s branch-2 passed with JDK v1.8.0_111
          +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.8.0_111
          +1 javac 0m 16s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.7.0_121
          +1 javac 0m 16s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 22s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 0m 41s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_111
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_121
          +1 unit 0m 19s hadoop-aws in the patch passed with JDK v1.7.0_121.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          28m 37s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13823
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840444/HADOOP-13823-branch-2-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 0c26c8df0a02 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 1edabc4
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11135/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11135/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 13m 37s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 9s branch-2 passed +1 compile 0m 18s branch-2 passed with JDK v1.8.0_111 +1 compile 0m 18s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 13s branch-2 passed +1 mvnsite 0m 23s branch-2 passed +1 mvneclipse 0m 20s branch-2 passed +1 findbugs 0m 34s branch-2 passed +1 javadoc 0m 13s branch-2 passed with JDK v1.8.0_111 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 18s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_111 +1 javac 0m 16s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_121 +1 javac 0m 16s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_111 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_121 +1 unit 0m 19s hadoop-aws in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 28m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13823 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840444/HADOOP-13823-branch-2-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 0c26c8df0a02 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 1edabc4 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11135/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11135/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          Committed to trunk through branch-2.8 branches. Thanks for your contribution, Steve Loughran.

          Show
          liuml07 Mingliang Liu added a comment - +1 Committed to trunk through branch-2.8 branches. Thanks for your contribution, Steve Loughran .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10901 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10901/)
          HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve (liuml07: rev d60a60be8aa450c44d3be69d26c88025e253ac0c)

          • (edit) hadoop-tools/hadoop-aws/src/test/resources/contract/s3a.xml
          • (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/RenameFailedException.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemContract.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10901 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10901/ ) HADOOP-13823 . s3a rename: fail if dest file exists. Contributed by Steve (liuml07: rev d60a60be8aa450c44d3be69d26c88025e253ac0c) (edit) hadoop-tools/hadoop-aws/src/test/resources/contract/s3a.xml (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/RenameFailedException.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemContract.java
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Steve Loughran You mentioned in description that you intend to have same behavior for s3a rename as HDFS. However, both HDFS and azure returns false when dest file already exists, but in this patch you are instead throwing exception in that case.
          Further, this changed semantics doesn't help Hive since even with this fix we need to handle HDFS and S3a differently because of return false vs throw exception. Was that intentional?

          Show
          ashutoshc Ashutosh Chauhan added a comment - Steve Loughran You mentioned in description that you intend to have same behavior for s3a rename as HDFS. However, both HDFS and azure returns false when dest file already exists, but in this patch you are instead throwing exception in that case. Further, this changed semantics doesn't help Hive since even with this fix we need to handle HDFS and S3a differently because of return false vs throw exception. Was that intentional?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm throwing an exception, from innerRename((): but I'm catching it in rename(). That RenameException() propagates the text and return code. It's intended to be used when we add a public rename/3, in which case the rename will forward the exception all the way up.

          Are you seeing, on a branch which has this patch in, an exception propagating? Or is your statement that "an exception is being raised" based on code review of the patch?

          Show
          stevel@apache.org Steve Loughran added a comment - I'm throwing an exception, from innerRename((): but I'm catching it in rename() . That RenameException() propagates the text and return code. It's intended to be used when we add a public rename/3, in which case the rename will forward the exception all the way up. Are you seeing, on a branch which has this patch in, an exception propagating? Or is your statement that "an exception is being raised" based on code review of the patch?

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development