Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13145

In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: tools/distcp
    • Labels:
      None
    • Target Version/s:

      Description

      After DistCp copies a file, it calls getFileStatus to get the FileStatus from the destination so that it can compare to the source and update metadata if necessary. If the DistCp command was run without the option to preserve metadata attributes, then this additional getFileStatus call is wasteful.

      1. HADOOP-13145.001.patch
        1 kB
        Chris Nauroth
      2. HADOOP-13145.003.patch
        18 kB
        Chris Nauroth
      3. HADOOP-13145-branch-2.004.patch
        17 kB
        Chris Nauroth
      4. HADOOP-13145-branch-2.8.004.patch
        17 kB
        Chris Nauroth

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          The attached v001 patch avoids the unnecessary getFileStatus call.

          The effect is particularly pronounced when running DistCp with a destination on S3A, where eventual consistency on S3 can cause the getFileStatus call to fail with FileNotFoundException. Then, the whole MapReduce task fails, retries, and repeats copying all the data. Rajesh Balamohan, I know you saw this with some recent large copies to S3A. Would you be interested in trying a test with this patch? So far, I don't have my own repro. Note that this patch is only helpful as long as the DistCp command is not preserving metadata attributes, so don't use the -p option.

          Cc Steve Loughran.

          Show
          cnauroth Chris Nauroth added a comment - The attached v001 patch avoids the unnecessary getFileStatus call. The effect is particularly pronounced when running DistCp with a destination on S3A, where eventual consistency on S3 can cause the getFileStatus call to fail with FileNotFoundException . Then, the whole MapReduce task fails, retries, and repeats copying all the data. Rajesh Balamohan , I know you saw this with some recent large copies to S3A. Would you be interested in trying a test with this patch? So far, I don't have my own repro. Note that this patch is only helpful as long as the DistCp command is not preserving metadata attributes, so don't use the -p option. Cc Steve Loughran .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 23s trunk passed
          +1 compile 0m 20s trunk passed with JDK v1.8.0_91
          +1 compile 0m 20s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 25s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 17s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 20s the patch passed
          +1 compile 0m 19s the patch passed with JDK v1.8.0_91
          +1 javac 0m 19s the patch passed
          +1 compile 0m 18s the patch passed with JDK v1.7.0_95
          +1 javac 0m 18s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 0m 15s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
          +1 unit 8m 49s hadoop-distcp in the patch passed with JDK v1.8.0_91.
          +1 unit 7m 52s hadoop-distcp in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          31m 31s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803972/HADOOP-13145.001.patch
          JIRA Issue HADOOP-13145
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6ad684579be4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3fa1380
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9420/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9420/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 23s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_91 +1 compile 0m 20s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 25s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 17s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 20s the patch passed +1 compile 0m 19s the patch passed with JDK v1.8.0_91 +1 javac 0m 19s the patch passed +1 compile 0m 18s the patch passed with JDK v1.7.0_95 +1 javac 0m 18s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 15s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 8m 49s hadoop-distcp in the patch passed with JDK v1.8.0_91. +1 unit 7m 52s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 31m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803972/HADOOP-13145.001.patch JIRA Issue HADOOP-13145 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6ad684579be4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fa1380 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9420/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9420/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Thanks for sharing the patch Chris Nauroth. I tried out the patch with hadoop-2.8 (may 16) and I do not see any more task failures with dist-cp to S3. I see some straggler tasks taking time, but that is completely unrelated to this jira.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Thanks for sharing the patch Chris Nauroth . I tried out the patch with hadoop-2.8 (may 16) and I do not see any more task failures with dist-cp to S3. I see some straggler tasks taking time, but that is completely unrelated to this jira.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          You know, I think s3a now has enough instrumentation that the # of times that getFileStatus is called would be measurable.

          At the very least, it'd be good to have a test of DistCp there, to verify that inconsistency problems aren't surfacing. The examples in, say TestDistCpViewFs , show a start, though I'd expect the new tests to simply throw up IOEs, rather than swallow + fail, the way that class does (and which I have just submitted a patch for, in HADOOP-13148).

          Show
          stevel@apache.org Steve Loughran added a comment - You know, I think s3a now has enough instrumentation that the # of times that getFileStatus is called would be measurable. At the very least, it'd be good to have a test of DistCp there, to verify that inconsistency problems aren't surfacing. The examples in, say TestDistCpViewFs , show a start, though I'd expect the new tests to simply throw up IOEs, rather than swallow + fail, the way that class does (and which I have just submitted a patch for, in HADOOP-13148 ).
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v003 adds a new abstract contract test suite for DistCp coverage and concrete test suite subclasses for S3A and WASB. I verified the tests are passing for both hadoop-aws (including running in parallel mode) and hadoop-azure.

          I'm going to leave the JIRA issue in Open status instead of Patch Available for now. The v003 patch will potentially hit Jenkins a little hard because of touching multiple modules, so I'd like to get another round of code review feedback first.

          Show
          cnauroth Chris Nauroth added a comment - Patch v003 adds a new abstract contract test suite for DistCp coverage and concrete test suite subclasses for S3A and WASB. I verified the tests are passing for both hadoop-aws (including running in parallel mode) and hadoop-azure. I'm going to leave the JIRA issue in Open status instead of Patch Available for now. The v003 patch will potentially hit Jenkins a little hard because of touching multiple modules, so I'd like to get another round of code review feedback first.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          tested -003 against s3 ireland and azure.

          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 223.843 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp
          
          ...
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.354 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
          
          

          Interesting how much faster azure is.

          The patch, is, as it stands, it's going to add 4 min to a TestS3A* test pattern. Could it be made one of the scaleable tests where it takes a config of option on scale so can be made configurable? There are already some tests which use scale.test.operation.count to control scale; we could have one on distcp file size, with the large file size being driven by it. Make it something in KB and it could easily be tuned for those of us in a different country from an S3 endpoint.

          Show
          stevel@apache.org Steve Loughran added a comment - tested -003 against s3 ireland and azure. Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 223.843 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp ... Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.354 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Interesting how much faster azure is. The patch, is, as it stands, it's going to add 4 min to a TestS3A* test pattern. Could it be made one of the scaleable tests where it takes a config of option on scale so can be made configurable? There are already some tests which use scale.test.operation.count to control scale; we could have one on distcp file size, with the large file size being driven by it. Make it something in KB and it could easily be tuned for those of us in a different country from an S3 endpoint.
          Hide
          cnauroth Chris Nauroth added a comment -

          Interestingly, you're getting a much slower run than me for S3A and a much faster run than me for WASB. I'm in the US Pacific Northwest. My S3 bucket is in US-west-2. My Azure Storage account is in West US.

          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 140.389 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
          
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 143.99 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp
          

          Could it be made one of the scaleable tests where it takes a config of option on scale so can be made configurable?

          We definitely could do that, but in my test runs, the large file tests don't show a significantly longer execution time. (See below for my timings.) Are the large file tests a long haul in your environment?

          Maybe a more effective change would be to cut down the number of test cases. I could keep just deepDirectoryStructureToRemote, largeFilesToRemote, deepDirectoryStructureFromRemote and largeFilesFromRemote. If I do that, then my S3A execution time comes down to 90 seconds. I don't think it sacrifices much in terms of coverage.

          Let me know your thoughts, and then I'll update the patch.

            <testcase name="multipleFilesToRemote" classname="org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time="22.084"/>
            <testcase name="deepDirectoryStructureFromRemote" classname="org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time="12.973"/>
            <testcase name="deepDirectoryStructureToRemote" classname="org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time="27.658"/>
            <testcase name="largeFilesToRemote" classname="org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time="26.381"/>
            <testcase name="singleFileToRemote" classname="org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time="12.197"/>
            <testcase name="largeFilesFromRemote" classname="org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time="18.894"/>
            <testcase name="multipleFilesFromRemote" classname="org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time="9.822"/>
            <testcase name="singleFileFromRemote" classname="org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time="6.835"/>
          
          Show
          cnauroth Chris Nauroth added a comment - Interestingly, you're getting a much slower run than me for S3A and a much faster run than me for WASB. I'm in the US Pacific Northwest. My S3 bucket is in US-west-2. My Azure Storage account is in West US. Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 140.389 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 143.99 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp Could it be made one of the scaleable tests where it takes a config of option on scale so can be made configurable? We definitely could do that, but in my test runs, the large file tests don't show a significantly longer execution time. (See below for my timings.) Are the large file tests a long haul in your environment? Maybe a more effective change would be to cut down the number of test cases. I could keep just deepDirectoryStructureToRemote , largeFilesToRemote , deepDirectoryStructureFromRemote and largeFilesFromRemote . If I do that, then my S3A execution time comes down to 90 seconds. I don't think it sacrifices much in terms of coverage. Let me know your thoughts, and then I'll update the patch. <testcase name= "multipleFilesToRemote" classname= "org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time= "22.084" /> <testcase name= "deepDirectoryStructureFromRemote" classname= "org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time= "12.973" /> <testcase name= "deepDirectoryStructureToRemote" classname= "org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time= "27.658" /> <testcase name= "largeFilesToRemote" classname= "org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time= "26.381" /> <testcase name= "singleFileToRemote" classname= "org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time= "12.197" /> <testcase name= "largeFilesFromRemote" classname= "org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time= "18.894" /> <testcase name= "multipleFilesFromRemote" classname= "org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time= "9.822" /> <testcase name= "singleFileFromRemote" classname= "org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp" time= "6.835" />
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 16s trunk passed
          +1 compile 6m 26s trunk passed
          +1 checkstyle 1m 20s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 57s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 13s trunk passed
          +1 javadoc 0m 46s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 0m 47s the patch passed
          +1 compile 6m 26s the patch passed
          +1 javac 6m 26s the patch passed
          -1 checkstyle 1m 20s root: The patch generated 1 new + 31 unchanged - 0 fixed = 32 total (was 31)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 30s the patch passed
          +1 javadoc 0m 46s the patch passed
          +1 unit 0m 9s hadoop-project in the patch passed.
          +1 unit 8m 11s hadoop-distcp in the patch passed.
          +1 unit 0m 13s hadoop-aws in the patch passed.
          +1 unit 1m 17s hadoop-azure in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          42m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804539/HADOOP-13145.003.patch
          JIRA Issue HADOOP-13145
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 971586d256d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 010e6ac
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9498/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9498/testReport/
          modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9498/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 16s trunk passed +1 compile 6m 26s trunk passed +1 checkstyle 1m 20s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 57s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 13s trunk passed +1 javadoc 0m 46s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 0m 47s the patch passed +1 compile 6m 26s the patch passed +1 javac 6m 26s the patch passed -1 checkstyle 1m 20s root: The patch generated 1 new + 31 unchanged - 0 fixed = 32 total (was 31) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 30s the patch passed +1 javadoc 0m 46s the patch passed +1 unit 0m 9s hadoop-project in the patch passed. +1 unit 8m 11s hadoop-distcp in the patch passed. +1 unit 0m 13s hadoop-aws in the patch passed. +1 unit 1m 17s hadoop-azure in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 42m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804539/HADOOP-13145.003.patch JIRA Issue HADOOP-13145 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 971586d256d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 010e6ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9498/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9498/testReport/ modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9498/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 35s Maven dependency ordering for branch
          +1 mvninstall 6m 36s trunk passed
          +1 compile 6m 44s trunk passed
          +1 checkstyle 1m 22s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 49s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 48s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 0m 50s the patch passed
          +1 compile 6m 39s the patch passed
          +1 javac 6m 39s the patch passed
          -1 checkstyle 1m 21s root: The patch generated 1 new + 31 unchanged - 0 fixed = 32 total (was 31)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 32s the patch passed
          +1 javadoc 0m 45s the patch passed
          +1 unit 0m 8s hadoop-project in the patch passed.
          +1 unit 8m 31s hadoop-distcp in the patch passed.
          +1 unit 0m 14s hadoop-aws in the patch passed.
          +1 unit 1m 20s hadoop-azure in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          44m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804539/HADOOP-13145.003.patch
          JIRA Issue HADOOP-13145
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 4c41cf5c1ba1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1597630
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9504/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9504/testReport/
          modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9504/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 35s Maven dependency ordering for branch +1 mvninstall 6m 36s trunk passed +1 compile 6m 44s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 49s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 12s trunk passed +1 javadoc 0m 48s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 0m 50s the patch passed +1 compile 6m 39s the patch passed +1 javac 6m 39s the patch passed -1 checkstyle 1m 21s root: The patch generated 1 new + 31 unchanged - 0 fixed = 32 total (was 31) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 32s the patch passed +1 javadoc 0m 45s the patch passed +1 unit 0m 8s hadoop-project in the patch passed. +1 unit 8m 31s hadoop-distcp in the patch passed. +1 unit 0m 14s hadoop-aws in the patch passed. +1 unit 1m 20s hadoop-azure in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 44m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804539/HADOOP-13145.003.patch JIRA Issue HADOOP-13145 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 4c41cf5c1ba1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9504/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9504/testReport/ modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9504/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          There's no S3 service in my country, I need to test against a datacentre in a country with a lower tax regime yet still under EU data protection legislation coverage. Ireland; I could benchmark Frankfurt.

          If you think the large files repeat the same coverage as the smaller ones, yes, please unify. Even so, I'd like it to be configurable so that I could set up test runs with smaller datasets —and we have the option of test runs with larger files.

          For those test, it'd be nice if the S3A setup explicitly turned the multipart threshold down (8MB?) and the same for partition sizes, so that it'd test the multipart code path and distcp

          Show
          stevel@apache.org Steve Loughran added a comment - There's no S3 service in my country, I need to test against a datacentre in a country with a lower tax regime yet still under EU data protection legislation coverage. Ireland; I could benchmark Frankfurt. If you think the large files repeat the same coverage as the smaller ones, yes, please unify. Even so, I'd like it to be configurable so that I could set up test runs with smaller datasets —and we have the option of test runs with larger files. For those test, it'd be nice if the S3A setup explicitly turned the multipart threshold down (8MB?) and the same for partition sizes, so that it'd test the multipart code path and distcp
          Hide
          stevel@apache.org Steve Loughran added a comment -

          BTW, Azure is in ireland too; the performance difference there is clearly not the pipe width and length at my end. Either it is S3 or it is how S3A talks to S3

          Show
          stevel@apache.org Steve Loughran added a comment - BTW, Azure is in ireland too; the performance difference there is clearly not the pipe width and length at my end. Either it is S3 or it is how S3A talks to S3
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 11s trunk passed
          +1 compile 7m 40s trunk passed
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 1m 11s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 49s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 7m 26s the patch passed
          +1 javac 7m 26s the patch passed
          -1 checkstyle 1m 32s root: The patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30)
          +1 mvnsite 1m 14s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 0m 48s the patch passed
          +1 unit 0m 9s hadoop-project in the patch passed.
          +1 unit 8m 36s hadoop-distcp in the patch passed.
          +1 unit 0m 15s hadoop-aws in the patch passed.
          +1 unit 1m 22s hadoop-azure in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          45m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804539/HADOOP-13145.003.patch
          JIRA Issue HADOOP-13145
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 462edf5d0168 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 141873c
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9520/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9520/testReport/
          modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9520/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 11s trunk passed +1 compile 7m 40s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 11s trunk passed +1 mvneclipse 0m 46s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 12s trunk passed +1 javadoc 0m 49s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 7m 26s the patch passed +1 javac 7m 26s the patch passed -1 checkstyle 1m 32s root: The patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 44s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 0m 9s hadoop-project in the patch passed. +1 unit 8m 36s hadoop-distcp in the patch passed. +1 unit 0m 15s hadoop-aws in the patch passed. +1 unit 1m 22s hadoop-azure in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 45m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804539/HADOOP-13145.003.patch JIRA Issue HADOOP-13145 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 462edf5d0168 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 141873c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9520/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9520/testReport/ modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9520/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching patch v004.

          • Removed redundant single-file tests and small multi-file tests.
          • Introduced scale.test.distcp.file.size.kb configuration property for tuning test file sizes. The default is 10 MB.
          • Set multi-part configuration properties to 8 MB, so with the default 10 MB file size, the tests will cover multi-part upload.

          With this version of the patch, the S3A test runs in ~55 seconds for me, and the WASB test runs in ~65 seconds. I completed a full parallel-test run against S3 buckets in US-west-2.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching patch v004. Removed redundant single-file tests and small multi-file tests. Introduced scale.test.distcp.file.size.kb configuration property for tuning test file sizes. The default is 10 MB. Set multi-part configuration properties to 8 MB, so with the default 10 MB file size, the tests will cover multi-part upload. With this version of the patch, the S3A test runs in ~55 seconds for me, and the WASB test runs in ~65 seconds. I completed a full parallel-test run against S3 buckets in US-west-2.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 35s Maven dependency ordering for branch
          +1 mvninstall 6m 35s branch-2 passed
          +1 compile 5m 35s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 23s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 23s branch-2 passed
          +1 mvnsite 1m 14s branch-2 passed
          +1 mvneclipse 0m 57s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 31s branch-2 passed
          +1 javadoc 0m 46s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 0m 54s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 5m 26s the patch passed with JDK v1.8.0_91
          +1 javac 5m 26s the patch passed
          +1 compile 6m 24s the patch passed with JDK v1.7.0_101
          +1 javac 6m 24s the patch passed
          -1 checkstyle 1m 24s root: The patch generated 2 new + 30 unchanged - 0 fixed = 32 total (was 30)
          +1 mvnsite 1m 15s the patch passed
          +1 mvneclipse 0m 52s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 0m 46s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 55s the patch passed with JDK v1.7.0_101
          +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.8.0_91.
          +1 unit 8m 34s hadoop-distcp in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_91.
          +1 unit 1m 19s hadoop-azure in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_101.
          +1 unit 7m 39s hadoop-distcp in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 unit 1m 42s hadoop-azure in the patch passed with JDK v1.7.0_101.
          -1 asflicense 0m 22s The patch generated 1 ASF License warnings.
          68m 23s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805165/HADOOP-13145-branch-2.004.patch
          JIRA Issue HADOOP-13145
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 6a4b05925028 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / e7f396f
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9533/artifact/patchprocess/diff-checkstyle-root.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9533/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/9533/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9533/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 35s Maven dependency ordering for branch +1 mvninstall 6m 35s branch-2 passed +1 compile 5m 35s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 23s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 23s branch-2 passed +1 mvnsite 1m 14s branch-2 passed +1 mvneclipse 0m 57s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 31s branch-2 passed +1 javadoc 0m 46s branch-2 passed with JDK v1.8.0_91 +1 javadoc 0m 54s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 5m 26s the patch passed with JDK v1.8.0_91 +1 javac 5m 26s the patch passed +1 compile 6m 24s the patch passed with JDK v1.7.0_101 +1 javac 6m 24s the patch passed -1 checkstyle 1m 24s root: The patch generated 2 new + 30 unchanged - 0 fixed = 32 total (was 30) +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 7s the patch passed +1 javadoc 0m 46s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 55s the patch passed with JDK v1.7.0_101 +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.8.0_91. +1 unit 8m 34s hadoop-distcp in the patch passed with JDK v1.8.0_91. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_91. +1 unit 1m 19s hadoop-azure in the patch passed with JDK v1.8.0_91. +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_101. +1 unit 7m 39s hadoop-distcp in the patch passed with JDK v1.7.0_101. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 unit 1m 42s hadoop-azure in the patch passed with JDK v1.7.0_101. -1 asflicense 0m 22s The patch generated 1 ASF License warnings. 68m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805165/HADOOP-13145-branch-2.004.patch JIRA Issue HADOOP-13145 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 6a4b05925028 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e7f396f Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9533/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9533/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/9533/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9533/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1

          latest patch brings test time down to <60s including all JUnit overhead.

          Thanks for doing this Chris, especially the tests. They'll be a good bit of regression testing in future

          Show
          stevel@apache.org Steve Loughran added a comment - +1 latest patch brings test time down to <60s including all JUnit overhead. Thanks for doing this Chris, especially the tests. They'll be a good bit of regression testing in future
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9832 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9832/)
          HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not (stevel: rev c918286b17e7f2a64735d0c972a8dd749e0bf6c4)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/TestS3AContractDistCp.java
          • hadoop-tools/hadoop-azure/pom.xml
          • hadoop-tools/hadoop-distcp/pom.xml
          • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractDistCp.java
          • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
          • hadoop-project/pom.xml
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9832 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9832/ ) HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not (stevel: rev c918286b17e7f2a64735d0c972a8dd749e0bf6c4) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/TestS3AContractDistCp.java hadoop-tools/hadoop-azure/pom.xml hadoop-tools/hadoop-distcp/pom.xml hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractDistCp.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java hadoop-project/pom.xml hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/pom.xml
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch as is breaks 2.8,

          
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-distcp: Compilation failure
          [ERROR] /Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:[80,25] cannot find symbol
          [ERROR] symbol:   method getTestDir()
          [ERROR] location: class org.apache.hadoop.test.GenericTestUtils
          [ERROR] -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          

          I've reverted the 2.8 patch; left it in branch-2+. Reopening so that a 2.8 specific patch can be added.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch as is breaks 2.8, [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile ( default -testCompile) on project hadoop-distcp: Compilation failure [ERROR] /Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:[80,25] cannot find symbol [ERROR] symbol: method getTestDir() [ERROR] location: class org.apache.hadoop.test.GenericTestUtils [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch . [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http: //cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command I've reverted the 2.8 patch; left it in branch-2+. Reopening so that a 2.8 specific patch can be added.
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve, thank you for catching the branch-2.8 problem and reverting. Sorry I didn't catch it myself earlier.

          I'm attaching a branch-2.8 patch. GenericTestUtils#getTestDir was introduced in your HADOOP-12984 patch, targeted to 2.9.0. That's a sizable patch, and I don't want to take on a back-port right now. Instead, this branch-2.8 patch goes back to the pre-HADOOP-12984 strategy of individual tests reading the test.build.data property directly.

          Show
          cnauroth Chris Nauroth added a comment - Steve, thank you for catching the branch-2.8 problem and reverting. Sorry I didn't catch it myself earlier. I'm attaching a branch-2.8 patch. GenericTestUtils#getTestDir was introduced in your HADOOP-12984 patch, targeted to 2.9.0. That's a sizable patch, and I don't want to take on a back-port right now. Instead, this branch-2.8 patch goes back to the pre- HADOOP-12984 strategy of individual tests reading the test.build.data property directly.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 2m 44s Maven dependency ordering for branch
          +1 mvninstall 7m 28s branch-2.8 passed
          +1 compile 6m 50s branch-2.8 passed with JDK v1.8.0_91
          +1 compile 7m 6s branch-2.8 passed with JDK v1.7.0_101
          +1 checkstyle 1m 12s branch-2.8 passed
          +1 mvnsite 1m 18s branch-2.8 passed
          +1 mvneclipse 1m 39s branch-2.8 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 43s branch-2.8 passed
          +1 javadoc 0m 50s branch-2.8 passed with JDK v1.8.0_91
          +1 javadoc 0m 52s branch-2.8 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 6m 54s the patch passed with JDK v1.8.0_91
          +1 javac 6m 54s the patch passed
          +1 compile 6m 57s the patch passed with JDK v1.7.0_101
          +1 javac 6m 57s the patch passed
          +1 checkstyle 1m 5s the patch passed
          +1 mvnsite 1m 17s the patch passed
          +1 mvneclipse 0m 52s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 0m 46s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 53s the patch passed with JDK v1.7.0_101
          +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_91.
          +1 unit 8m 17s hadoop-distcp in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_91.
          +1 unit 1m 26s hadoop-azure in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_101.
          +1 unit 7m 28s hadoop-distcp in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 unit 1m 40s hadoop-azure in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          75m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c60792e
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805258/HADOOP-13145-branch-2.8.004.patch
          JIRA Issue HADOOP-13145
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 9af7054e2a79 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.8 / ce17c9f
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9540/testReport/
          modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9540/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 2m 44s Maven dependency ordering for branch +1 mvninstall 7m 28s branch-2.8 passed +1 compile 6m 50s branch-2.8 passed with JDK v1.8.0_91 +1 compile 7m 6s branch-2.8 passed with JDK v1.7.0_101 +1 checkstyle 1m 12s branch-2.8 passed +1 mvnsite 1m 18s branch-2.8 passed +1 mvneclipse 1m 39s branch-2.8 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 43s branch-2.8 passed +1 javadoc 0m 50s branch-2.8 passed with JDK v1.8.0_91 +1 javadoc 0m 52s branch-2.8 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 6m 54s the patch passed with JDK v1.8.0_91 +1 javac 6m 54s the patch passed +1 compile 6m 57s the patch passed with JDK v1.7.0_101 +1 javac 6m 57s the patch passed +1 checkstyle 1m 5s the patch passed +1 mvnsite 1m 17s the patch passed +1 mvneclipse 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 6s the patch passed +1 javadoc 0m 46s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 53s the patch passed with JDK v1.7.0_101 +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_91. +1 unit 8m 17s hadoop-distcp in the patch passed with JDK v1.8.0_91. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_91. +1 unit 1m 26s hadoop-azure in the patch passed with JDK v1.8.0_91. +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_101. +1 unit 7m 28s hadoop-distcp in the patch passed with JDK v1.7.0_101. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 unit 1m 40s hadoop-azure in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 75m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:c60792e JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805258/HADOOP-13145-branch-2.8.004.patch JIRA Issue HADOOP-13145 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 9af7054e2a79 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / ce17c9f Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9540/testReport/ modules C: hadoop-project hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9540/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1

          applied the branch-2.8 patch, tested, all is well.

          Show
          stevel@apache.org Steve Loughran added a comment - +1 applied the branch-2.8 patch, tested, all is well.
          Hide
          adamjk Adam Kramer added a comment -

          Any chance of creating a patch/applying to 2.7 branch?

          Show
          adamjk Adam Kramer added a comment - Any chance of creating a patch/applying to 2.7 branch?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          we're generally pretty reluctant to put stuff into 2.7.x which isn't a fairly significant bug fix, just because every change is at a risk of breaking something, and we like 2.7.x to be stable. There's enough change in this patch (poms, code, new tests) that it's got the potential to cause trouble

          Hadoop 2.8.1 is/will be out in the next few days, could you work with that?

          Show
          stevel@apache.org Steve Loughran added a comment - we're generally pretty reluctant to put stuff into 2.7.x which isn't a fairly significant bug fix, just because every change is at a risk of breaking something, and we like 2.7.x to be stable. There's enough change in this patch (poms, code, new tests) that it's got the potential to cause trouble Hadoop 2.8.1 is/will be out in the next few days, could you work with that?
          Hide
          adamjk Adam Kramer added a comment -

          We're using Spark that is pre-built to 2.7 but I can try building Spark against 2.8.1 when it's released to see how it goes.

          Show
          adamjk Adam Kramer added a comment - We're using Spark that is pre-built to 2.7 but I can try building Spark against 2.8.1 when it's released to see how it goes.
          Hide
          adamjk Adam Kramer added a comment -

          Steve Loughran Any idea when 2.8.1 will be released?

          Show
          adamjk Adam Kramer added a comment - Steve Loughran Any idea when 2.8.1 will be released?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          a 2.81. did come out, but security related. Go with 2.8.0 for now

          Show
          stevel@apache.org Steve Loughran added a comment - a 2.81. did come out, but security related. Go with 2.8.0 for now

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development