Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13660

Upgrade commons-configuration version to 2.1

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Bump commons-configuration version from 1.6 to 2.1

      Description

      We're currently pulling in version 1.6 - I think we should upgrade to the latest 1.10.

      1. HADOOP-13660.001.patch
        0.4 kB
        Sean Mackrory
      2. HADOOP-13660.002.patch
        29 kB
        Sean Mackrory
      3. HADOOP-13660.003.patch
        28 kB
        Sean Mackrory
      4. HADOOP-13660.004.patch
        28 kB
        Sean Mackrory
      5. HADOOP-13660.005.patch
        28 kB
        Sean Mackrory
      6. HADOOP-13660.006.patch
        29 kB
        Sean Mackrory
      7. HADOOP-13660.007.patch
        29 kB
        Sean Mackrory
      8. HADOOP-13660-configuration2.001.patch
        26 kB
        Sean Mackrory

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 6s trunk passed
          +1 compile 0m 8s trunk passed
          +1 mvnsite 0m 11s trunk passed
          +1 mvneclipse 0m 10s trunk passed
          +1 javadoc 0m 9s trunk passed
          +1 mvninstall 0m 8s the patch passed
          +1 compile 0m 7s the patch passed
          +1 javac 0m 7s the patch passed
          +1 mvnsite 0m 9s the patch passed
          +1 mvneclipse 0m 7s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 7s the patch passed
          +1 unit 0m 6s hadoop-project in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          10m 43s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13660
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830411/HADOOP-13660.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux bb3d45308dd9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8e06d86
          Default Java 1.8.0_101
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10600/testReport/
          modules C: hadoop-project U: hadoop-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10600/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 6s trunk passed +1 compile 0m 8s trunk passed +1 mvnsite 0m 11s trunk passed +1 mvneclipse 0m 10s trunk passed +1 javadoc 0m 9s trunk passed +1 mvninstall 0m 8s the patch passed +1 compile 0m 7s the patch passed +1 javac 0m 7s the patch passed +1 mvnsite 0m 9s the patch passed +1 mvneclipse 0m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 7s the patch passed +1 unit 0m 6s hadoop-project in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 10m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13660 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830411/HADOOP-13660.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux bb3d45308dd9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8e06d86 Default Java 1.8.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10600/testReport/ modules C: hadoop-project U: hadoop-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10600/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I am aware that Whirr, Accumulo uses commons-configuration.

          Accumulo specifies its version, 1.6.
          Whirr specifies its version, 1.7.

          Since both does not inherit version from Hadoop, it should be fine unless there are other projects that inherit commons-configuration version from Hadoop.

          So I am +1 for this patch, and I'd like to commit this patch by end of this week. Please shout out if any one has any concerns.

          Show
          jojochuang Wei-Chiu Chuang added a comment - I am aware that Whirr, Accumulo uses commons-configuration. Accumulo specifies its version, 1.6. Whirr specifies its version, 1.7. Since both does not inherit version from Hadoop, it should be fine unless there are other projects that inherit commons-configuration version from Hadoop. So I am +1 for this patch, and I'd like to commit this patch by end of this week. Please shout out if any one has any concerns.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          So I do have one extra comment: the last 1.x commons-configuration release, 1.10, was released in 2013 October. Should we consider migrating to 2.1 where there seems to be more active development? My concern is that if we do not migrate now, we have to wait until next Hadoop major release, and by then it might be 7 years since the last 1.x release of commons-configuration.

          However, given that this dependency is used lightly in Hadoop codebase and I am not aware of downstream applications relying on Hadoop's commons-configuration version, it may be fine if we have to migrate to commons-configuration 2.x in a Hadoop 3.x minor release.

          Show
          jojochuang Wei-Chiu Chuang added a comment - So I do have one extra comment: the last 1.x commons-configuration release, 1.10, was released in 2013 October. Should we consider migrating to 2.1 where there seems to be more active development? My concern is that if we do not migrate now, we have to wait until next Hadoop major release, and by then it might be 7 years since the last 1.x release of commons-configuration. However, given that this dependency is used lightly in Hadoop codebase and I am not aware of downstream applications relying on Hadoop's commons-configuration version, it may be fine if we have to migrate to commons-configuration 2.x in a Hadoop 3.x minor release.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Thanks Wei-Chiu Chuang. I'm taking a look at what it would take to move to 2.x. They've done the release in a way that allows 1.x and 2.x to coexist nicely, but I agree we might as well update now if we reasonably can.

          Show
          mackrorysd Sean Mackrory added a comment - Thanks Wei-Chiu Chuang . I'm taking a look at what it would take to move to 2.x. They've done the release in a way that allows 1.x and 2.x to coexist nicely, but I agree we might as well update now if we reasonably can.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Just as an update, attaching what I've been working on. This is after working through the migration guide and linked documentation. This compiles and should be reasonably close to how it was before. Currently TestMetricsConfig is failing and it appears the root problem is a difference in how config files are located: some files that used to be found automatically now can't be found. Strategies for location files are no configurable, but even configuring it to try every strategy, it's still failing to find files that previous tests seemed to find just fine. Will continue looking at it...

          Show
          mackrorysd Sean Mackrory added a comment - Just as an update, attaching what I've been working on. This is after working through the migration guide and linked documentation. This compiles and should be reasonably close to how it was before. Currently TestMetricsConfig is failing and it appears the root problem is a difference in how config files are located: some files that used to be found automatically now can't be found. Strategies for location files are no configurable, but even configuring it to try every strategy, it's still failing to find files that previous tests seemed to find just fine. Will continue looking at it...
          Hide
          mackrorysd Sean Mackrory added a comment -

          Got to the bottom of the test failures and addressed them. I believe all tests are working now without any significant modifications to the tests themselves. Still needs a bit more testing, but my confidence is increasing in this migration...

          Show
          mackrorysd Sean Mackrory added a comment - Got to the bottom of the test failures and addressed them. I believe all tests are working now without any significant modifications to the tests themselves. Still needs a bit more testing, but my confidence is increasing in this migration...
          Hide
          mackrorysd Sean Mackrory added a comment -

          Tweaking the way commons-beanutils is declared in the POM, otherwise it's missing in the classpath for a lot of tests. I think it's really a dependency to use this, not just to run the tests, but isn't found at compile-time.

          All tests definitely passing locally (except those failing before anyway), and I've done some manual testing to be sure.

          Show
          mackrorysd Sean Mackrory added a comment - Tweaking the way commons-beanutils is declared in the POM, otherwise it's missing in the classpath for a lot of tests. I think it's really a dependency to use this, not just to run the tests, but isn't found at compile-time. All tests definitely passing locally (except those failing before anyway), and I've done some manual testing to be sure.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hey Sean Mackrory thanks a lot for putting these together. It mostly looks good with just a few nits:

          • IOException is imported but not used in MetricsConfig.java
          • MetricsConfig#create
                    // Commons Configuration defines the message text when file not found
                    if (e.getMessage().startsWith("Could not locate")) {
                      continue;
                    }
            

            Depending on exception message is generally risky. Would it be possible to add a regression test? Or is it already covered by existing unit tests?

          • This conversion is a little bit concerning:
            conf.setListDelimiterHandler(new DefaultListDelimiterHandler(','));
            

            According to https://commons.apache.org/proper/commons-configuration/userguide/upgradeto2_0.html#Accessing_Configuration_Properties DefaultListDelimiterHandler is not 100% compatible with the old behavior (even though the old one is inconsistent). Instead, LegacyListDelimiterHandler is said to preserve old behavior. Want to call this out so we can study it further.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hey Sean Mackrory thanks a lot for putting these together. It mostly looks good with just a few nits: IOException is imported but not used in MetricsConfig.java MetricsConfig#create // Commons Configuration defines the message text when file not found if (e.getMessage().startsWith( "Could not locate" )) { continue ; } Depending on exception message is generally risky. Would it be possible to add a regression test? Or is it already covered by existing unit tests? This conversion is a little bit concerning: conf.setListDelimiterHandler( new DefaultListDelimiterHandler(',')); According to https://commons.apache.org/proper/commons-configuration/userguide/upgradeto2_0.html#Accessing_Configuration_Properties DefaultListDelimiterHandler is not 100% compatible with the old behavior (even though the old one is inconsistent). Instead, LegacyListDelimiterHandler is said to preserve old behavior. Want to call this out so we can study it further.
          Hide
          mackrorysd Sean Mackrory added a comment - - edited

          IOException is imported but not used in MetricsConfig.java

          Fixed. Also, we're able to use a helper class that makes the new code in this class a little more concise (fresh patch attached).

          Depending on exception message is generally risky. Would it be possible to add a regression test? Or is it already covered by existing unit tests?

          This is already covered by existing unit tests: TestMetricsConfig#testLoadFirst and TestMetricsConfig#testMissingFiles will both fail if the code doesn't gracefully detect that one of the files it attempts to load are missing and proceeds to try elsewhere.

          Want to call this out so we can study it further.

          You mean like a release note? Can definitely add one. I believe the concern is very minor here, though. I've compared the logic for reading in the configuration, and the code is functionally equivalent (it's actually mostly identical). The documented definition of how things get escaped in a list also remains the same. Where things differ is exactly how things get escaped when writing back out the configuration, which appears to mainly be done for logging / debugging. Even then, it should only be different for some edge cases that as far as I can tell should come up exceptionally rarely (perhaps never) in the Metrics2 use case (values that end with backslashes, etc.) Tough to be sure I've looked at all the use cases exhaustively, though.

          Show
          mackrorysd Sean Mackrory added a comment - - edited IOException is imported but not used in MetricsConfig.java Fixed. Also, we're able to use a helper class that makes the new code in this class a little more concise (fresh patch attached). Depending on exception message is generally risky. Would it be possible to add a regression test? Or is it already covered by existing unit tests? This is already covered by existing unit tests: TestMetricsConfig#testLoadFirst and TestMetricsConfig#testMissingFiles will both fail if the code doesn't gracefully detect that one of the files it attempts to load are missing and proceeds to try elsewhere. Want to call this out so we can study it further. You mean like a release note? Can definitely add one. I believe the concern is very minor here, though. I've compared the logic for reading in the configuration, and the code is functionally equivalent (it's actually mostly identical). The documented definition of how things get escaped in a list also remains the same. Where things differ is exactly how things get escaped when writing back out the configuration, which appears to mainly be done for logging / debugging. Even then, it should only be different for some edge cases that as far as I can tell should come up exceptionally rarely (perhaps never) in the Metrics2 use case (values that end with backslashes, etc.) Tough to be sure I've looked at all the use cases exhaustively, though.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks for taking the extra care and the clarification, Sean!
          I'll review the latest patch again today.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks for taking the extra care and the clarification, Sean! I'll review the latest patch again today.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 54s trunk passed
          +1 compile 6m 52s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 3m 10s trunk passed
          +1 mvneclipse 1m 17s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 5m 16s trunk passed
          +1 javadoc 2m 28s trunk passed
          0 mvndep 0m 22s Maven dependency ordering for patch
          +1 mvninstall 2m 42s the patch passed
          +1 compile 6m 49s the patch passed
          +1 javac 6m 49s the patch passed
          -0 checkstyle 1m 35s root: The patch generated 7 new + 374 unchanged - 8 fixed = 381 total (was 382)
          +1 mvnsite 3m 26s the patch passed
          +1 mvneclipse 1m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          -1 findbugs 1m 35s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 2m 55s the patch passed
          +1 unit 0m 14s hadoop-project in the patch passed.
          +1 unit 9m 13s hadoop-common in the patch passed.
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          -1 unit 54m 29s hadoop-hdfs in the patch failed.
          -1 unit 1m 13s hadoop-azure in the patch failed.
          +1 unit 0m 20s hadoop-kafka in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          145m 32s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Found reliance on default encoding in org.apache.hadoop.metrics2.impl.MetricsConfig.toString(Configuration):in org.apache.hadoop.metrics2.impl.MetricsConfig.toString(Configuration): new java.io.PrintWriter(OutputStream) At MetricsConfig.java:[line 283]
          Failed junit tests hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
            hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
            hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
            hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
            hadoop.fs.azure.TestFileSystemOperationExceptionHandling
            hadoop.fs.azure.TestWasbFsck
            hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
            hadoop.fs.azure.TestOutOfBandAzureBlobOperations
            hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
            hadoop.fs.azure.TestFileSystemOperationExceptionMessage
            hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
            hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
            hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
            hadoop.fs.azure.TestContainerChecks
            hadoop.fs.azure.TestNativeAzureFileSystemLive
            hadoop.fs.azure.TestNativeAzureFileSystemContractLive
            hadoop.fs.azure.TestBlobMetadata
            hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
            hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
            hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
            hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
            hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
            hadoop.fs.azure.TestNativeAzureFSPageBlobLive
            hadoop.fs.azure.TestBlobDataValidation
            hadoop.fs.azure.TestBlobTypeSpeedDifference
            hadoop.fs.azure.TestWasbUriAndConfiguration
            hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
            hadoop.fs.azure.TestNativeAzureFileSystemAppend
            hadoop.fs.azure.TestNativeAzureFileSystemMocked
            hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
            hadoop.fs.azure.TestAzureFileSystemErrorConditions
            hadoop.fs.azure.TestFileSystemOperationsWithThreads
          Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
            org.apache.hadoop.hdfs.TestFileChecksum
            org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
            org.apache.hadoop.hdfs.security.token.block.TestBlockToken
            org.apache.hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            org.apache.hadoop.hdfs.security.TestDelegationToken
            org.apache.hadoop.hdfs.TestLeaseRecovery
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.cli.TestErasureCodingCLI
            org.apache.hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13660
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835149/HADOOP-13660.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 005a0144c638 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbd2057
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 54s trunk passed +1 compile 6m 52s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 3m 10s trunk passed +1 mvneclipse 1m 17s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 5m 16s trunk passed +1 javadoc 2m 28s trunk passed 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 2m 42s the patch passed +1 compile 6m 49s the patch passed +1 javac 6m 49s the patch passed -0 checkstyle 1m 35s root: The patch generated 7 new + 374 unchanged - 8 fixed = 381 total (was 382) +1 mvnsite 3m 26s the patch passed +1 mvneclipse 1m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project -1 findbugs 1m 35s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 55s the patch passed +1 unit 0m 14s hadoop-project in the patch passed. +1 unit 9m 13s hadoop-common in the patch passed. +1 unit 1m 9s hadoop-hdfs-client in the patch passed. -1 unit 54m 29s hadoop-hdfs in the patch failed. -1 unit 1m 13s hadoop-azure in the patch failed. +1 unit 0m 20s hadoop-kafka in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 145m 32s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Found reliance on default encoding in org.apache.hadoop.metrics2.impl.MetricsConfig.toString(Configuration):in org.apache.hadoop.metrics2.impl.MetricsConfig.toString(Configuration): new java.io.PrintWriter(OutputStream) At MetricsConfig.java: [line 283] Failed junit tests hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded   hadoop.fs.azure.TestNativeAzureFileSystemConcurrency   hadoop.fs.azure.TestAzureConcurrentOutOfBandIo   hadoop.fs.azure.TestFileSystemOperationExceptionHandling   hadoop.fs.azure.TestWasbFsck   hadoop.fs.azure.TestNativeAzureFileSystemClientLogging   hadoop.fs.azure.TestOutOfBandAzureBlobOperations   hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite   hadoop.fs.azure.TestFileSystemOperationExceptionMessage   hadoop.fs.azure.TestNativeAzureFileSystemContractMocked   hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive   hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive   hadoop.fs.azure.TestContainerChecks   hadoop.fs.azure.TestNativeAzureFileSystemLive   hadoop.fs.azure.TestNativeAzureFileSystemContractLive   hadoop.fs.azure.TestBlobMetadata   hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation   hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked   hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem   hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator   hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList   hadoop.fs.azure.TestNativeAzureFSPageBlobLive   hadoop.fs.azure.TestBlobDataValidation   hadoop.fs.azure.TestBlobTypeSpeedDifference   hadoop.fs.azure.TestWasbUriAndConfiguration   hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck   hadoop.fs.azure.TestNativeAzureFileSystemAppend   hadoop.fs.azure.TestNativeAzureFileSystemMocked   hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations   hadoop.fs.azure.TestAzureFileSystemErrorConditions   hadoop.fs.azure.TestFileSystemOperationsWithThreads Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead   org.apache.hadoop.hdfs.TestFileChecksum   org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery   org.apache.hadoop.hdfs.security.token.block.TestBlockToken   org.apache.hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   org.apache.hadoop.hdfs.security.TestDelegationToken   org.apache.hadoop.hdfs.TestLeaseRecovery   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.cli.TestErasureCodingCLI   org.apache.hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13660 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835149/HADOOP-13660.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 005a0144c638 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbd2057 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10892/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment - - edited

          I think the azure test failures are unrelated. Saw them in https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/135/ too. But other test failures should be looked at.

          Show
          jojochuang Wei-Chiu Chuang added a comment - - edited I think the azure test failures are unrelated. Saw them in https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/135/ too. But other test failures should be looked at.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Thanks Wei-Chiu Chuang. I think the hadoop-hdfs test failures are also unrelated. I would be very surprised if my changes impacted the test that failed, and I can't reproduce it locally now. Let's see if it comes up this time...

          I also took a look at the findbugs issue. The PrintStream instance there is no longer necessary - we can go from the PrintWriter that configuration2 requires directly to the ByteArrayOutputStream to get a string (changed in the latest patch). I suspect it might still complain about me not providing a charset to the PrintWriter - I'm not sure it's worth what is required to work around all the requirements here, though. This function is only used for generating log messages. Where a charset can be specified, one is.

          Show
          mackrorysd Sean Mackrory added a comment - Thanks Wei-Chiu Chuang . I think the hadoop-hdfs test failures are also unrelated. I would be very surprised if my changes impacted the test that failed, and I can't reproduce it locally now. Let's see if it comes up this time... I also took a look at the findbugs issue. The PrintStream instance there is no longer necessary - we can go from the PrintWriter that configuration2 requires directly to the ByteArrayOutputStream to get a string (changed in the latest patch). I suspect it might still complain about me not providing a charset to the PrintWriter - I'm not sure it's worth what is required to work around all the requirements here, though. This function is only used for generating log messages. Where a charset can be specified, one is.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          The previous test failures were originated from HADOOP-13514, which has since been reverted. So let's see if the new patch passes these tests.

          Show
          jojochuang Wei-Chiu Chuang added a comment - The previous test failures were originated from HADOOP-13514 , which has since been reverted. So let's see if the new patch passes these tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 8m 11s trunk passed
          +1 compile 7m 49s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 3m 18s trunk passed
          +1 mvneclipse 1m 20s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 5m 55s trunk passed
          +1 javadoc 2m 35s trunk passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 2m 50s the patch passed
          +1 compile 7m 15s the patch passed
          +1 javac 7m 15s the patch passed
          -0 checkstyle 1m 37s root: The patch generated 7 new + 374 unchanged - 8 fixed = 381 total (was 382)
          +1 mvnsite 3m 36s the patch passed
          +1 mvneclipse 1m 37s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          -1 findbugs 1m 38s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 2m 47s the patch passed
          +1 unit 0m 15s hadoop-project in the patch passed.
          +1 unit 8m 13s hadoop-common in the patch passed.
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
          +1 unit 60m 0s hadoop-hdfs in the patch passed.
          -1 unit 1m 30s hadoop-azure in the patch failed.
          +1 unit 0m 20s hadoop-kafka in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          153m 50s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Found reliance on default encoding in org.apache.hadoop.metrics2.impl.MetricsConfig.toString(Configuration):in org.apache.hadoop.metrics2.impl.MetricsConfig.toString(Configuration): new java.io.PrintWriter(OutputStream, boolean) At MetricsConfig.java:[line 279]
          Failed junit tests hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13660
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835847/HADOOP-13660.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux a3896ef9bb3c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1b79c41
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 11s trunk passed +1 compile 7m 49s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 3m 18s trunk passed +1 mvneclipse 1m 20s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 5m 55s trunk passed +1 javadoc 2m 35s trunk passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 2m 50s the patch passed +1 compile 7m 15s the patch passed +1 javac 7m 15s the patch passed -0 checkstyle 1m 37s root: The patch generated 7 new + 374 unchanged - 8 fixed = 381 total (was 382) +1 mvnsite 3m 36s the patch passed +1 mvneclipse 1m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project -1 findbugs 1m 38s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 47s the patch passed +1 unit 0m 15s hadoop-project in the patch passed. +1 unit 8m 13s hadoop-common in the patch passed. +1 unit 1m 2s hadoop-hdfs-client in the patch passed. +1 unit 60m 0s hadoop-hdfs in the patch passed. -1 unit 1m 30s hadoop-azure in the patch failed. +1 unit 0m 20s hadoop-kafka in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 153m 50s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Found reliance on default encoding in org.apache.hadoop.metrics2.impl.MetricsConfig.toString(Configuration):in org.apache.hadoop.metrics2.impl.MetricsConfig.toString(Configuration): new java.io.PrintWriter(OutputStream, boolean) At MetricsConfig.java: [line 279] Failed junit tests hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13660 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835847/HADOOP-13660.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux a3896ef9bb3c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1b79c41 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10917/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Sean Mackrory thanks a lot again for your new patch. I think we should either address the findbug issue, or exclude it from the check if we think that's a false positive. Either way we can't let that findbug warning stay there.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Sean Mackrory thanks a lot again for your new patch. I think we should either address the findbug issue, or exclude it from the check if we think that's a false positive. Either way we can't let that findbug warning stay there.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Added a findbugs exclusion for the error, for the reasons listed above. No other changes from .005 to .006.

          Show
          mackrorysd Sean Mackrory added a comment - Added a findbugs exclusion for the error, for the reasons listed above. No other changes from .005 to .006.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 8m 29s trunk passed
          +1 compile 8m 2s trunk passed
          +1 checkstyle 1m 38s trunk passed
          +1 mvnsite 3m 43s trunk passed
          +1 mvneclipse 1m 21s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 5m 54s trunk passed
          +1 javadoc 2m 39s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 44s the patch passed
          +1 compile 7m 47s the patch passed
          +1 javac 7m 47s the patch passed
          -0 checkstyle 1m 39s root: The patch generated 7 new + 374 unchanged - 8 fixed = 381 total (was 382)
          +1 mvnsite 3m 58s the patch passed
          +1 mvneclipse 1m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 7m 43s the patch passed
          +1 javadoc 2m 47s the patch passed
          +1 unit 0m 14s hadoop-project in the patch passed.
          +1 unit 8m 21s hadoop-common in the patch passed.
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
          -1 unit 62m 17s hadoop-hdfs in the patch failed.
          +1 unit 1m 31s hadoop-azure in the patch passed.
          +1 unit 0m 20s hadoop-kafka in the patch passed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          159m 3s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestStartup



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13660
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836366/HADOOP-13660.006.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7bde4936cd37 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / de01327
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10996/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10996/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10996/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10996/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 8m 29s trunk passed +1 compile 8m 2s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 3m 43s trunk passed +1 mvneclipse 1m 21s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 5m 54s trunk passed +1 javadoc 2m 39s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 44s the patch passed +1 compile 7m 47s the patch passed +1 javac 7m 47s the patch passed -0 checkstyle 1m 39s root: The patch generated 7 new + 374 unchanged - 8 fixed = 381 total (was 382) +1 mvnsite 3m 58s the patch passed +1 mvneclipse 1m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 7m 43s the patch passed +1 javadoc 2m 47s the patch passed +1 unit 0m 14s hadoop-project in the patch passed. +1 unit 8m 21s hadoop-common in the patch passed. +1 unit 1m 2s hadoop-hdfs-client in the patch passed. -1 unit 62m 17s hadoop-hdfs in the patch failed. +1 unit 1m 31s hadoop-azure in the patch passed. +1 unit 0m 20s hadoop-kafka in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 159m 3s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13660 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836366/HADOOP-13660.006.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7bde4936cd37 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / de01327 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10996/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10996/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10996/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10996/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 9m 21s trunk passed
          +1 compile 12m 18s trunk passed
          +1 checkstyle 1m 46s trunk passed
          +1 mvnsite 4m 27s trunk passed
          +1 mvneclipse 2m 8s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 51s trunk passed
          +1 javadoc 3m 26s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 3m 4s the patch passed
          +1 compile 9m 40s the patch passed
          +1 javac 9m 40s the patch passed
          -0 checkstyle 1m 50s root: The patch generated 7 new + 374 unchanged - 8 fixed = 381 total (was 382)
          +1 mvnsite 4m 45s the patch passed
          +1 mvneclipse 2m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 7m 41s the patch passed
          +1 javadoc 3m 47s the patch passed
          +1 unit 0m 21s hadoop-project in the patch passed.
          +1 unit 9m 10s hadoop-common in the patch passed.
          +1 unit 1m 24s hadoop-hdfs-client in the patch passed.
          -1 unit 59m 14s hadoop-hdfs in the patch failed.
          +1 unit 1m 47s hadoop-azure in the patch passed.
          +1 unit 0m 30s hadoop-kafka in the patch passed.
          +1 asflicense 0m 52s The patch does not generate ASF License warnings.
          172m 28s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.TestCrcCorruption



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13660
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836366/HADOOP-13660.006.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b933f5da59ec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e1c6ef2
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11038/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11038/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11038/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11038/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 9m 21s trunk passed +1 compile 12m 18s trunk passed +1 checkstyle 1m 46s trunk passed +1 mvnsite 4m 27s trunk passed +1 mvneclipse 2m 8s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 51s trunk passed +1 javadoc 3m 26s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 3m 4s the patch passed +1 compile 9m 40s the patch passed +1 javac 9m 40s the patch passed -0 checkstyle 1m 50s root: The patch generated 7 new + 374 unchanged - 8 fixed = 381 total (was 382) +1 mvnsite 4m 45s the patch passed +1 mvneclipse 2m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 7m 41s the patch passed +1 javadoc 3m 47s the patch passed +1 unit 0m 21s hadoop-project in the patch passed. +1 unit 9m 10s hadoop-common in the patch passed. +1 unit 1m 24s hadoop-hdfs-client in the patch passed. -1 unit 59m 14s hadoop-hdfs in the patch failed. +1 unit 1m 47s hadoop-azure in the patch passed. +1 unit 0m 30s hadoop-kafka in the patch passed. +1 asflicense 0m 52s The patch does not generate ASF License warnings. 172m 28s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13660 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836366/HADOOP-13660.006.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b933f5da59ec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e1c6ef2 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11038/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11038/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11038/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11038/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          mackrorysd Sean Mackrory added a comment -

          FWIW I don't see these test failures locally and I'm fairly certain they're completely unrelated.

          Show
          mackrorysd Sean Mackrory added a comment - FWIW I don't see these test failures locally and I'm fairly certain they're completely unrelated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hi Sean Mackrory thanks a lot for the new patch and I think it's almost good. Would you also like to fix checkstyle warning?

          ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/PositionStripeReader.java:21:import org.apache.commons.configuration2.SystemConfiguration;:8: Unused import - org.apache.commons.configuration2.SystemConfiguration.

          +1 after fixing this checkstyle warning.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hi Sean Mackrory thanks a lot for the new patch and I think it's almost good. Would you also like to fix checkstyle warning? ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/PositionStripeReader.java:21:import org.apache.commons.configuration2.SystemConfiguration;:8: Unused import - org.apache.commons.configuration2.SystemConfiguration. +1 after fixing this checkstyle warning.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Ah yes - fixed in .007. That file was part of the automated change in the package name - must have been unused before, too.

          Show
          mackrorysd Sean Mackrory added a comment - Ah yes - fixed in .007. That file was part of the automated change in the package name - must have been unused before, too.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 22s trunk passed
          +1 compile 9m 49s trunk passed
          +1 checkstyle 1m 43s trunk passed
          +1 mvnsite 3m 49s trunk passed
          +1 mvneclipse 1m 53s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 4s trunk passed
          +1 javadoc 3m 7s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 2m 50s the patch passed
          +1 compile 9m 38s the patch passed
          +1 javac 9m 38s the patch passed
          -0 checkstyle 1m 42s root: The patch generated 6 new + 374 unchanged - 8 fixed = 380 total (was 382)
          +1 mvnsite 4m 9s the patch passed
          +1 mvneclipse 2m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 7m 19s the patch passed
          +1 javadoc 3m 23s the patch passed
          +1 unit 0m 19s hadoop-project in the patch passed.
          +1 unit 8m 45s hadoop-common in the patch passed.
          -1 unit 0m 34s hadoop-hdfs-client in the patch failed.
          -1 unit 0m 45s hadoop-hdfs in the patch failed.
          -1 unit 0m 39s hadoop-azure in the patch failed.
          -1 unit 0m 22s hadoop-kafka in the patch failed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          119m 42s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13660
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839246/HADOOP-13660.007.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 479f47bdada8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 59bfcbf
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-kafka.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 22s trunk passed +1 compile 9m 49s trunk passed +1 checkstyle 1m 43s trunk passed +1 mvnsite 3m 49s trunk passed +1 mvneclipse 1m 53s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 4s trunk passed +1 javadoc 3m 7s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 50s the patch passed +1 compile 9m 38s the patch passed +1 javac 9m 38s the patch passed -0 checkstyle 1m 42s root: The patch generated 6 new + 374 unchanged - 8 fixed = 380 total (was 382) +1 mvnsite 4m 9s the patch passed +1 mvneclipse 2m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 7m 19s the patch passed +1 javadoc 3m 23s the patch passed +1 unit 0m 19s hadoop-project in the patch passed. +1 unit 8m 45s hadoop-common in the patch passed. -1 unit 0m 34s hadoop-hdfs-client in the patch failed. -1 unit 0m 45s hadoop-hdfs in the patch failed. -1 unit 0m 39s hadoop-azure in the patch failed. -1 unit 0m 22s hadoop-kafka in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 119m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13660 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839246/HADOOP-13660.007.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 479f47bdada8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 59bfcbf Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-kafka.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11080/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hmmm not sure what caused the compilation error. My local build didn't fail. I am triggering the precommit build again.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hmmm not sure what caused the compilation error. My local build didn't fail. I am triggering the precommit build again.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
          0 mvndep 2m 3s Maven dependency ordering for branch
          +1 mvninstall 7m 21s trunk passed
          +1 compile 9m 23s trunk passed
          +1 checkstyle 1m 36s trunk passed
          +1 mvnsite 3m 52s trunk passed
          +1 mvneclipse 1m 45s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 5m 56s trunk passed
          +1 javadoc 3m 4s trunk passed
          0 mvndep 0m 23s Maven dependency ordering for patch
          +1 mvninstall 2m 38s the patch passed
          +1 compile 9m 7s the patch passed
          +1 javac 9m 7s the patch passed
          -0 checkstyle 1m 39s root: The patch generated 6 new + 375 unchanged - 8 fixed = 381 total (was 383)
          +1 mvnsite 3m 57s the patch passed
          +1 mvneclipse 2m 7s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 32s the patch passed
          +1 javadoc 3m 15s the patch passed
          +1 unit 0m 20s hadoop-project in the patch passed.
          +1 unit 7m 21s hadoop-common in the patch passed.
          +1 unit 1m 5s hadoop-hdfs-client in the patch passed.
          -1 unit 64m 54s hadoop-hdfs in the patch failed.
          +1 unit 1m 35s hadoop-azure in the patch passed.
          +1 unit 0m 25s hadoop-kafka in the patch passed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          166m 2s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13660
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839246/HADOOP-13660.007.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 85bf031a6f7d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ce2847e
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11084/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11084/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11084/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11084/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 2m 3s Maven dependency ordering for branch +1 mvninstall 7m 21s trunk passed +1 compile 9m 23s trunk passed +1 checkstyle 1m 36s trunk passed +1 mvnsite 3m 52s trunk passed +1 mvneclipse 1m 45s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 5m 56s trunk passed +1 javadoc 3m 4s trunk passed 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 2m 38s the patch passed +1 compile 9m 7s the patch passed +1 javac 9m 7s the patch passed -0 checkstyle 1m 39s root: The patch generated 6 new + 375 unchanged - 8 fixed = 381 total (was 383) +1 mvnsite 3m 57s the patch passed +1 mvneclipse 2m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 32s the patch passed +1 javadoc 3m 15s the patch passed +1 unit 0m 20s hadoop-project in the patch passed. +1 unit 7m 21s hadoop-common in the patch passed. +1 unit 1m 5s hadoop-hdfs-client in the patch passed. -1 unit 64m 54s hadoop-hdfs in the patch failed. +1 unit 1m 35s hadoop-azure in the patch passed. +1 unit 0m 25s hadoop-kafka in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 166m 2s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13660 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839246/HADOOP-13660.007.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 85bf031a6f7d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ce2847e Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11084/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11084/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11084/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-kafka U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11084/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          +1. The failed test is not reproducible in my local machine.

          Show
          jojochuang Wei-Chiu Chuang added a comment - +1. The failed test is not reproducible in my local machine.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed the 007 patch to trunk. Thanks Sean Mackrory for the great contribution!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed the 007 patch to trunk. Thanks Sean Mackrory for the great contribution!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10861 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10861/)
          HADOOP-13660. Upgrade commons-configuration version. Contributed by Sean (weichiu: rev c0b1a44f6c6e6f9e4ac5cecea0d4a50e237a4c9c)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/StatsDSink.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
          • (edit) hadoop-project/pom.xml
          • (edit) hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/ganglia/TestGangliaSink.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/ConfigUtil.java
          • (edit) hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsConfig.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/GangliaSink30.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/ConfigBuilder.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsCollectorImpl.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSink.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/filter/AbstractPatternFilter.java
          • (edit) hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/FileSink.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/AbstractGangliaSink.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
          • (edit) hadoop-tools/hadoop-azure/pom.xml
          • (edit) hadoop-common-project/hadoop-common/pom.xml
          • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/filter/TestPatternFilter.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/PositionStripeReader.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeFSDataSetSink.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsPlugin.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10861 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10861/ ) HADOOP-13660 . Upgrade commons-configuration version. Contributed by Sean (weichiu: rev c0b1a44f6c6e6f9e4ac5cecea0d4a50e237a4c9c) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/StatsDSink.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java (edit) hadoop-project/pom.xml (edit) hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/ganglia/TestGangliaSink.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/ConfigUtil.java (edit) hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsConfig.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/GangliaSink30.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/ConfigBuilder.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsCollectorImpl.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSink.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/filter/AbstractPatternFilter.java (edit) hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/FileSink.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/AbstractGangliaSink.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java (edit) hadoop-tools/hadoop-azure/pom.xml (edit) hadoop-common-project/hadoop-common/pom.xml (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/filter/TestPatternFilter.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/PositionStripeReader.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeFSDataSetSink.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsPlugin.java
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm staring at a build thing elsewhere related to this; convergency errors related to commons-lang3 and that of azure and dynamodb.

          1. what was the reason for excluding commons-lang3 as a dependency of azure in the azure project? I'm not faulting the decision, just trying to understand what's happening.
          2. is everyone confident that all azure storage codepaths are valid without that dependency?
          3. I don't see any explicit declaration on a commons-lang version. Should we have one? Because s3guard is pulling in dynamodb local for testing, and I'm getting some (local) convergence issues?
          +-org.apache.hadoop:hadoop-aws
            +-com.amazonaws:DynamoDBLocal:1.11.0
              +-org.apache.commons:commons-lang3:3.3.2
          

          This isn't surfacing in the ASF branch because of the exclusion, I'm just wondering what to do.

          FWIW, I'm thinking of declaring lang3 dependency of org.apache.commons:commons-lang3:jar:3.4 in the hadoop-project pom, and including it in test scope for hadoop-aws+s3guard, while excludingng the dynamodb one, it being older

          Show
          stevel@apache.org Steve Loughran added a comment - I'm staring at a build thing elsewhere related to this; convergency errors related to commons-lang3 and that of azure and dynamodb. what was the reason for excluding commons-lang3 as a dependency of azure in the azure project? I'm not faulting the decision, just trying to understand what's happening. is everyone confident that all azure storage codepaths are valid without that dependency? I don't see any explicit declaration on a commons-lang version. Should we have one? Because s3guard is pulling in dynamodb local for testing, and I'm getting some (local) convergence issues? +-org.apache.hadoop:hadoop-aws +-com.amazonaws:DynamoDBLocal:1.11.0 +-org.apache.commons:commons-lang3:3.3.2 This isn't surfacing in the ASF branch because of the exclusion, I'm just wondering what to do. FWIW, I'm thinking of declaring lang3 dependency of org.apache.commons:commons-lang3:jar:3.4 in the hadoop-project pom, and including it in test scope for hadoop-aws+s3guard, while excludingng the dynamodb one, it being older
          Hide
          mackrorysd Sean Mackrory added a comment -

          I don't recall but I just took a quick look. I'll bet what happened is I ran into the convergence issue in Azure (because as you said, that comes up if you remove the exclusion), and I excluded it and meant to go back and version-manage that more properly once everything else was working.

          All tests pass for me now, and would have back then, but that's as close as I get to being confident that all azure storage codepaths are valid without that dependency. I'd be all for properly declaring the version we depend on in the hadoop-project pom as you're suggesting...

          Show
          mackrorysd Sean Mackrory added a comment - I don't recall but I just took a quick look. I'll bet what happened is I ran into the convergence issue in Azure (because as you said, that comes up if you remove the exclusion), and I excluded it and meant to go back and version-manage that more properly once everything else was working. All tests pass for me now, and would have back then, but that's as close as I get to being confident that all azure storage codepaths are valid without that dependency. I'd be all for properly declaring the version we depend on in the hadoop-project pom as you're suggesting...
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks...what I will do is have a patch for s3guard which also fixes up the dynamo reference; this will need to go into trunk without that bit of the patch.

          Testing things locally now in HADOOP-13994

          Show
          stevel@apache.org Steve Loughran added a comment - thanks...what I will do is have a patch for s3guard which also fixes up the dynamo reference; this will need to go into trunk without that bit of the patch. Testing things locally now in HADOOP-13994
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I just realized this changed public API signature too, so flag it as "Incompatible change"

          The MetricsPlugin interface API MetricsPlugin.init() has a SubsetConfiguration parameter. The parameter changed from org.apache.commons.configuration.SubsetConfiguration to org.apache.commons.configuration2.SubsetConfiguration. So anything depending on this public API will need to be recompiled at least.

          For example, HadoopTimelineMetricsSink in Ambari breaks because of this change.

          Show
          jojochuang Wei-Chiu Chuang added a comment - I just realized this changed public API signature too, so flag it as "Incompatible change" The MetricsPlugin interface API MetricsPlugin.init() has a SubsetConfiguration parameter. The parameter changed from org.apache.commons.configuration.SubsetConfiguration to org.apache.commons.configuration2.SubsetConfiguration. So anything depending on this public API will need to be recompiled at least. For example, HadoopTimelineMetricsSink in Ambari breaks because of this change.

            People

            • Assignee:
              mackrorysd Sean Mackrory
              Reporter:
              mackrorysd Sean Mackrory
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development