Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13406

S3AFileSystem: Consider reusing filestatus in delete() and mkdirs()

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      filestatus can be reused in rename() and in mkdirs.

      1. HADOOP-13406-branch-2-001.patch
        4 kB
        Rajesh Balamohan
      2. HADOOP-13406-branch-2-002.patch
        3 kB
        Steve Loughran
      3. HADOOP-13406-branch-2-003.patch
        3 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          S3 test results are given below. Failures are not related to this patch.

          Results :
          
          Tests in error:
            TestS3AContractDistCp>AbstractFSContractTestBase.teardown:190->AbstractFSContractTestBase.deleteTestDirInTeardown:198 »
            TestS3AContractDistCp>AbstractContractDistCpTest.largeFilesFromRemote:108->AbstractContractDistCpTest.largeFiles:174 » FileNotFound
            TestS3ADeleteFilesOneByOne>TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 »
            TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 »  test timed out after 1800...
            TestS3ABlockingThreadPool.testFastMultiPartUpload:76 » FileNotFound Multi-part...
            TestS3ADirectoryPerformance.testTimeToStatNonEmptyDirectory:153->timeToStatPath:179 »
          
          Tests run: 263, Failures: 0, Errors: 5, Skipped: 7
          

          Please note that this may have to be rebased (if needed) when HADOOP-13208 is committed.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - S3 test results are given below. Failures are not related to this patch. Results : Tests in error: TestS3AContractDistCp>AbstractFSContractTestBase.teardown:190->AbstractFSContractTestBase.deleteTestDirInTeardown:198 » TestS3AContractDistCp>AbstractContractDistCpTest.largeFilesFromRemote:108->AbstractContractDistCpTest.largeFiles:174 » FileNotFound TestS3ADeleteFilesOneByOne>TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 » TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 » test timed out after 1800... TestS3ABlockingThreadPool.testFastMultiPartUpload:76 » FileNotFound Multi-part... TestS3ADirectoryPerformance.testTimeToStatNonEmptyDirectory:153->timeToStatPath:179 » Tests run: 263, Failures: 0, Errors: 5, Skipped: 7 Please note that this may have to be rebased (if needed) when HADOOP-13208 is committed.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          what infra are you testing against that's so unreliable here? Have you tried setting the fs.s3a.endpoint to the specific location?

          Show
          stevel@apache.org Steve Loughran added a comment - what infra are you testing against that's so unreliable here? Have you tried setting the fs.s3a.endpoint to the specific location?
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          This is from my local laptop connecting to west coast. May be i should switch to ap-southeast-1 and try it out.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - This is from my local laptop connecting to west coast. May be i should switch to ap-southeast-1 and try it out.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I like the performance boost: 1-2 extra getFileStatus calls saved, leading to 4+ HTTP requests saved on a file copy operation.

          Patch-wise -1 as is. Adds a bit of convolution to the code, with a new innerDelete. I've done a rework which I'll see what Jenkins thinks of. This reduces the diff between the branch and your patch more (no new innerDelete), changes the javadocs.

          Show
          stevel@apache.org Steve Loughran added a comment - I like the performance boost: 1-2 extra getFileStatus calls saved, leading to 4+ HTTP requests saved on a file copy operation. Patch-wise -1 as is. Adds a bit of convolution to the code, with a new innerDelete. I've done a rework which I'll see what Jenkins thinks of. This reduces the diff between the branch and your patch more (no new innerDelete), changes the javadocs.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 002

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 002
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 15m 1s branch-2 passed
          +1 compile 0m 13s branch-2 passed with JDK v1.8.0_91
          +1 compile 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 17s branch-2 passed
          +1 mvnsite 0m 22s branch-2 passed
          +1 mvneclipse 1m 25s branch-2 passed
          +1 findbugs 0m 40s branch-2 passed
          +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 0m 16s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 10s the patch passed with JDK v1.8.0_91
          +1 javac 0m 10s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_101
          +1 javac 0m 13s the patch passed
          -0 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 41s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_101
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          23m 50s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819639/HADOOP-13406-branch-2-002.patch
          JIRA Issue HADOOP-13406
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 79def062a230 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / ea10e13
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10064/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10064/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10064/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 1s branch-2 passed +1 compile 0m 13s branch-2 passed with JDK v1.8.0_91 +1 compile 0m 15s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 17s branch-2 passed +1 mvnsite 0m 22s branch-2 passed +1 mvneclipse 1m 25s branch-2 passed +1 findbugs 0m 40s branch-2 passed +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_91 +1 javadoc 0m 16s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 14s the patch passed +1 compile 0m 10s the patch passed with JDK v1.8.0_91 +1 javac 0m 10s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_101 +1 javac 0m 13s the patch passed -0 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_101 +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 23m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819639/HADOOP-13406-branch-2-002.patch JIRA Issue HADOOP-13406 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 79def062a230 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / ea10e13 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10064/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10064/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10064/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 003; checkstyle

          BTW, I still consider this rajesh's patch; I've only cleaned it up. So I am prepared to +1 it myself, once jenkins green lights it

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 003; checkstyle BTW, I still consider this rajesh's patch; I've only cleaned it up. So I am prepared to +1 it myself, once jenkins green lights it
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 24s branch-2 passed
          +1 compile 0m 13s branch-2 passed with JDK v1.8.0_91
          +1 compile 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 15s branch-2 passed
          +1 mvnsite 0m 20s branch-2 passed
          +1 mvneclipse 0m 13s branch-2 passed
          +1 findbugs 0m 32s branch-2 passed
          +1 javadoc 0m 12s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 10s the patch passed with JDK v1.8.0_91
          +1 javac 0m 10s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_101
          +1 javac 0m 13s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 42s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_101
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          13m 45s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819789/HADOOP-13406-branch-2-003.patch
          JIRA Issue HADOOP-13406
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7de7b7df1f36 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / bdde7ed
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10069/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10069/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 24s branch-2 passed +1 compile 0m 13s branch-2 passed with JDK v1.8.0_91 +1 compile 0m 15s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 15s branch-2 passed +1 mvnsite 0m 20s branch-2 passed +1 mvneclipse 0m 13s branch-2 passed +1 findbugs 0m 32s branch-2 passed +1 javadoc 0m 12s branch-2 passed with JDK v1.8.0_91 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 14s the patch passed +1 compile 0m 10s the patch passed with JDK v1.8.0_91 +1 javac 0m 10s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_101 +1 javac 0m 13s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_101 +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 13m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819789/HADOOP-13406-branch-2-003.patch JIRA Issue HADOOP-13406 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7de7b7df1f36 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / bdde7ed Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10069/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10069/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1. Committed. Thanks!

          Show
          stevel@apache.org Steve Loughran added a comment - +1. Committed. Thanks!
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-trunk-Commit #10143 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10143/)
          HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and (stevel: rev be9e46b42dd1ed0b2295bd36a7d81d5ee6dffc25)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-trunk-Commit #10143 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10143/ ) HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and (stevel: rev be9e46b42dd1ed0b2295bd36a7d81d5ee6dffc25) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

            People

            • Assignee:
              rajesh.balamohan Rajesh Balamohan
              Reporter:
              rajesh.balamohan Rajesh Balamohan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development