Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10544

Balancer doesn't work with IPFailoverProxyProvider

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Right now Balancer gets the NN URIs through DFSUtil#getNameServiceUris, which returns logical URIs in HA is enabled. If IPFailoverProxyProvider is used, Balancer will not be able to start.

      I think the bug is at DFSUtil#getNameServiceUris:

          for (String nsId : getNameServiceIds(conf)) {
            if (HAUtil.isHAEnabled(conf, nsId)) {
              // Add the logical URI of the nameservice.
              try {
                ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
      

      Then if clause should also consider if the FailoverProxyProvider has useLogicalURI enabled. If not, getNameServiceUris should try to resolve the physical URI for this nsId.

      1. HDFS-10544.00.patch
        0.8 kB
        Zhe Zhang
      2. HDFS-10544.01.patch
        4 kB
        Zhe Zhang
      3. HDFS-10544.02.patch
        1 kB
        Zhe Zhang
      4. HDFS-10544.03.patch
        3 kB
        Zhe Zhang
      5. HDFS-10544.04.patch
        7 kB
        Zhe Zhang
      6. HDFS-10544.05.patch
        7 kB
        Zhe Zhang
      7. HDFS-10544-branch-2.7.patch
        8 kB
        Zhe Zhang

        Activity

        Hide
        zhz Zhe Zhang added a comment -

        Submitting initial patch.

        HAUtil#useLogicalUri can potentially be optimized to avoid creating an instance of AbstractNNFailoverProxyProvider.

        Show
        zhz Zhe Zhang added a comment - Submitting initial patch. HAUtil#useLogicalUri can potentially be optimized to avoid creating an instance of AbstractNNFailoverProxyProvider .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 27s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 34s trunk passed
        +1 compile 1m 5s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 1m 12s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 50s trunk passed
        +1 javadoc 0m 57s trunk passed
        -1 mvninstall 0m 33s hadoop-hdfs in the patch failed.
        -1 compile 0m 38s hadoop-hdfs in the patch failed.
        -1 javac 0m 38s hadoop-hdfs in the patch failed.
        +1 checkstyle 0m 29s the patch passed
        -1 mvnsite 0m 40s hadoop-hdfs in the patch failed.
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 20s hadoop-hdfs in the patch failed.
        +1 javadoc 1m 0s the patch passed
        -1 unit 0m 40s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        20m 13s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811433/HDFS-10544.00.patch
        JIRA Issue HDFS-10544
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 375e2b323f2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2800695
        Default Java 1.8.0_91
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
        mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15811/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15811/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 34s trunk passed +1 compile 1m 5s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 57s trunk passed -1 mvninstall 0m 33s hadoop-hdfs in the patch failed. -1 compile 0m 38s hadoop-hdfs in the patch failed. -1 javac 0m 38s hadoop-hdfs in the patch failed. +1 checkstyle 0m 29s the patch passed -1 mvnsite 0m 40s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 20s hadoop-hdfs in the patch failed. +1 javadoc 1m 0s the patch passed -1 unit 0m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 20m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811433/HDFS-10544.00.patch JIRA Issue HDFS-10544 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 375e2b323f2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2800695 Default Java 1.8.0_91 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15811/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15811/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15811/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        v0 patch didn't work because HAUtil#useLogicalUri expects an URI rather than String (btw sorry I missed the build failure before uploading patch). I don't think this is a good API design, because nameservice ID can be an arbitrary string.

        But to avoid changing public API HAUtil#useLogicalUri, in v1 patch I added another API to get the boolean condition from string. Also using the static method to check, rather than creating an instance of the proxy provider.

        Show
        zhz Zhe Zhang added a comment - v0 patch didn't work because HAUtil#useLogicalUri expects an URI rather than String (btw sorry I missed the build failure before uploading patch). I don't think this is a good API design, because nameservice ID can be an arbitrary string. But to avoid changing public API HAUtil#useLogicalUri , in v1 patch I added another API to get the boolean condition from string. Also using the static method to check, rather than creating an instance of the proxy provider.
        Hide
        zhz Zhe Zhang added a comment -

        Uploading v2 patch. So we have 2 alternatives:

        • v1 patch: add useLogicalUri to take String as parameter
        • v2 patch: throw IllegalArgumentException before trying to determine whether to use logical URI.

        I'm hesitating between the 2 options because the documentation is not completely clear about dfs.nameservices. Could it be an arbitrary string? If that arbitrary string fails to form a legal URI (when prefixed with hdfs://), shall we treat it just as a Runtime exception, or we should look for that arbitrary string prefixed with dfs.namenode.servicerpc-address in the config?

        Any suggestions are very welcome.

        Show
        zhz Zhe Zhang added a comment - Uploading v2 patch. So we have 2 alternatives: v1 patch: add useLogicalUri to take String as parameter v2 patch: throw IllegalArgumentException before trying to determine whether to use logical URI. I'm hesitating between the 2 options because the documentation is not completely clear about dfs.nameservices . Could it be an arbitrary string? If that arbitrary string fails to form a legal URI (when prefixed with hdfs:// ), shall we treat it just as a Runtime exception, or we should look for that arbitrary string prefixed with dfs.namenode.servicerpc-address in the config? Any suggestions are very welcome.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 46s trunk passed
        +1 compile 0m 49s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 44s trunk passed
        +1 javadoc 0m 55s trunk passed
        +1 mvninstall 0m 50s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 78 unchanged - 0 fixed = 80 total (was 78)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 0m 55s the patch passed
        -1 unit 62m 32s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        82m 6s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs
          Dead store to provider in org.apache.hadoop.hdfs.HAUtil.useLogicalUri(Configuration, String) At HAUtil.java:org.apache.hadoop.hdfs.HAUtil.useLogicalUri(Configuration, String) At HAUtil.java:[line 282]
        Failed junit tests hadoop.hdfs.TestAsyncHDFSWithHA



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811472/HDFS-10544.01.patch
        JIRA Issue HDFS-10544
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2152bb126f09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2800695
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15816/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15816/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15816/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15816/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15816/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15816/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 46s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 78 unchanged - 0 fixed = 80 total (was 78) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 55s the patch passed -1 unit 62m 32s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 82m 6s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Dead store to provider in org.apache.hadoop.hdfs.HAUtil.useLogicalUri(Configuration, String) At HAUtil.java:org.apache.hadoop.hdfs.HAUtil.useLogicalUri(Configuration, String) At HAUtil.java: [line 282] Failed junit tests hadoop.hdfs.TestAsyncHDFSWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811472/HDFS-10544.01.patch JIRA Issue HDFS-10544 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2152bb126f09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2800695 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15816/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15816/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/15816/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15816/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15816/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15816/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 15s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 0m 56s trunk passed
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 47s the patch passed
        +1 javac 0m 47s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 53s the patch passed
        -1 unit 61m 55s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        82m 17s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSUtil



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811475/HDFS-10544.02.patch
        JIRA Issue HDFS-10544
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8dffcec55b50 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2800695
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15817/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15817/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15817/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15817/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 15s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 61m 55s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 82m 17s Reason Tests Failed junit tests hadoop.hdfs.TestDFSUtil Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811475/HDFS-10544.02.patch JIRA Issue HDFS-10544 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8dffcec55b50 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2800695 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15817/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15817/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15817/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15817/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thought more about the two options. I think we should use the simpler approach of throwing IllegalArgumentException before checking if the proxy provider works with logical Uris. This is because no matter if the config uses ConfiguredFailoverProxyProvider or IPFailoverProxyProvider, a legal URI is needed by the constructor. So it's actually a valid requirement that nsId can form a legal URI.

        Also updating the unit test TestDFSUtil. Basically, 2 of the tests were using logical URIs without specifying a FailoverProxyProvider. With such a config, the correct behavior of DFSUtil#getInternalNameServices is to try to resolve the logical URI, instead of waiting for the proxy provider to do so. But in those 2 tests, the config doesn't have the correct dfs.namenode.servicerpc-address entries to resolve the logical URIs.

        Show
        zhz Zhe Zhang added a comment - Thought more about the two options. I think we should use the simpler approach of throwing IllegalArgumentException before checking if the proxy provider works with logical Uris. This is because no matter if the config uses ConfiguredFailoverProxyProvider or IPFailoverProxyProvider , a legal URI is needed by the constructor. So it's actually a valid requirement that nsId can form a legal URI. Also updating the unit test TestDFSUtil . Basically, 2 of the tests were using logical URIs without specifying a FailoverProxyProvider . With such a config, the correct behavior of DFSUtil#getInternalNameServices is to try to resolve the logical URI, instead of waiting for the proxy provider to do so. But in those 2 tests, the config doesn't have the correct dfs.namenode.servicerpc-address entries to resolve the logical URIs.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 46s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 11s trunk passed
        +1 findbugs 1m 45s trunk passed
        +1 javadoc 0m 54s trunk passed
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        -1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 135 unchanged - 0 fixed = 137 total (was 135)
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 48s the patch passed
        +1 javadoc 0m 55s the patch passed
        -1 unit 65m 9s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        84m 18s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.hdfs.server.namenode.TestCacheDirectives
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
        Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812321/HDFS-10544.03.patch
        JIRA Issue HDFS-10544
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f8f3003d51e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d8107fc
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15859/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15859/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15859/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15859/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15859/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 135 unchanged - 0 fixed = 137 total (was 135) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 55s the patch passed -1 unit 65m 9s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 84m 18s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812321/HDFS-10544.03.patch JIRA Issue HDFS-10544 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f8f3003d51e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8107fc Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15859/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15859/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15859/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15859/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15859/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        shv Konstantin Shvachko added a comment -

        Looking at the test case you added to TestDFSUtil.testGetNNUris(). Here is the essential part of the configuration:

        fs.defaultFS=hdfs://nn2.example.com:9820, 
        dfs.namenode.rpc-address=hdfs://nn.example.com:9820,
        dfs.nameservices=ns1,ns2,
        dfs.namenode.servicerpc-address.ns2=ns2-nn.example.com:9820,
        dfs.ha.namenodes.ns1=nn1,nn2,
        dfs.namenode.rpc-address.ns1.nn1=ns1-nn1.example.com:9820, 
        dfs.namenode.rpc-address.ns1.nn2=ns1-nn2.example.com:9820, 
        dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.IPFailoverProxyProvider,
        

        And DFSUtil.getInternalNsRpcUris() returns uris = [hdfs://ns2-nn.example.com:9820, hdfs://nn.example.com:9820]. Not sure if this is what you expected.

        Also in the test we know exactly what those URIs returned by getInternalNsRpcUris() should be, so it would make sense to add asserts for the values in addition to checking the number of the URIs.

        Show
        shv Konstantin Shvachko added a comment - Looking at the test case you added to TestDFSUtil.testGetNNUris() . Here is the essential part of the configuration: fs.defaultFS=hdfs: //nn2.example.com:9820, dfs.namenode.rpc-address=hdfs: //nn.example.com:9820, dfs.nameservices=ns1,ns2, dfs.namenode.servicerpc-address.ns2=ns2-nn.example.com:9820, dfs.ha.namenodes.ns1=nn1,nn2, dfs.namenode.rpc-address.ns1.nn1=ns1-nn1.example.com:9820, dfs.namenode.rpc-address.ns1.nn2=ns1-nn2.example.com:9820, dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.IPFailoverProxyProvider, And DFSUtil.getInternalNsRpcUris() returns uris = [hdfs://ns2-nn.example.com:9820, hdfs://nn.example.com:9820] . Not sure if this is what you expected. Also in the test we know exactly what those URIs returned by getInternalNsRpcUris() should be, so it would make sense to add asserts for the values in addition to checking the number of the URIs.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Konstantin Shvachko for the review! Updating the patch to address the comments.

        ... Not sure if this is what you expected.

        I updated the test to set a config entry to map ns1 to a physical URI. So the else statement in DFSUtil#getNameServiceUris will be able to resolve ns1 to a physical URI. I think this is the correct configuration in an environment that uses IPFailoverProxyProvider.

        Also updated the comments. Please kindly review.

        Show
        zhz Zhe Zhang added a comment - Thanks Konstantin Shvachko for the review! Updating the patch to address the comments. ... Not sure if this is what you expected. I updated the test to set a config entry to map ns1 to a physical URI. So the else statement in DFSUtil#getNameServiceUris will be able to resolve ns1 to a physical URI. I think this is the correct configuration in an environment that uses IPFailoverProxyProvider . Also updated the comments. Please kindly review.
        Hide
        shv Konstantin Shvachko added a comment -

        +1
        I can see the test failing without the patch and succeeds with the patch.

        Show
        shv Konstantin Shvachko added a comment - +1 I can see the test failing without the patch and succeeds with the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 59s trunk passed
        +1 compile 0m 55s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 1s trunk passed
        +1 javadoc 1m 0s trunk passed
        +1 mvninstall 1m 4s the patch passed
        +1 compile 1m 3s the patch passed
        +1 javac 1m 3s the patch passed
        -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 135 unchanged - 0 fixed = 139 total (was 135)
        +1 mvnsite 1m 4s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 19s the patch passed
        +1 javadoc 1m 4s the patch passed
        -1 unit 66m 19s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        91m 21s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817279/HDFS-10544.04.patch
        JIRA Issue HDFS-10544
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d94e914d3bf9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a290a98
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16037/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16037/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16037/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16037/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 59s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 0s trunk passed +1 mvninstall 1m 4s the patch passed +1 compile 1m 3s the patch passed +1 javac 1m 3s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 135 unchanged - 0 fixed = 139 total (was 135) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 1m 4s the patch passed -1 unit 66m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 91m 21s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817279/HDFS-10544.04.patch JIRA Issue HDFS-10544 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d94e914d3bf9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a290a98 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16037/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16037/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16037/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16037/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Konstantin Shvachko for the review! Attaching v5 patch which fixes 2 checkstyle issues in v4 (lines in TestDFSUtil too long). The other 2 checkstyle issues are inherent with the original code style.

        The reported test failures are unrelated and cannot be reproduced locally.

        I'll commit v5 patch shortly.

        Show
        zhz Zhe Zhang added a comment - Thanks Konstantin Shvachko for the review! Attaching v5 patch which fixes 2 checkstyle issues in v4 (lines in TestDFSUtil too long). The other 2 checkstyle issues are inherent with the original code style. The reported test failures are unrelated and cannot be reproduced locally. I'll commit v5 patch shortly.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10086 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10086/)
        HDFS-10544. Balancer doesn't work with IPFailoverProxyProvider. (zhz: rev 087290e6b1cb1082646d966b65494082712ebe3e)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10086 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10086/ ) HDFS-10544 . Balancer doesn't work with IPFailoverProxyProvider. (zhz: rev 087290e6b1cb1082646d966b65494082712ebe3e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
        Hide
        zhz Zhe Zhang added a comment -

        Committed v5 patch to trunk, branch-2, and branch-2.8. I'm working on resolving branch-2.7 conflicts.

        Show
        zhz Zhe Zhang added a comment - Committed v5 patch to trunk, branch-2, and branch-2.8. I'm working on resolving branch-2.7 conflicts.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 29s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 45s trunk passed
        +1 javadoc 0m 55s trunk passed
        +1 mvninstall 0m 54s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 135 unchanged - 0 fixed = 137 total (was 135)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 54s the patch passed
        +1 javadoc 0m 56s the patch passed
        +1 unit 59m 52s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        80m 11s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817602/HDFS-10544.05.patch
        JIRA Issue HDFS-10544
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2d63fcb45c0a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 06c56ff
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16039/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16039/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16039/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 29s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 135 unchanged - 0 fixed = 137 total (was 135) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 59m 52s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 80m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817602/HDFS-10544.05.patch JIRA Issue HDFS-10544 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2d63fcb45c0a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 06c56ff Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16039/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16039/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16039/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Attaching branch-2.7 patch to trigger Jenkins.

        Show
        zhz Zhe Zhang added a comment - Attaching branch-2.7 patch to trigger Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 11m 28s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 12s branch-2.7 passed
        +1 compile 1m 1s branch-2.7 passed with JDK v1.8.0_91
        +1 compile 1m 2s branch-2.7 passed with JDK v1.7.0_101
        +1 checkstyle 0m 24s branch-2.7 passed
        +1 mvnsite 1m 3s branch-2.7 passed
        +1 mvneclipse 0m 16s branch-2.7 passed
        +1 findbugs 3m 8s branch-2.7 passed
        +1 javadoc 1m 10s branch-2.7 passed with JDK v1.8.0_91
        +1 javadoc 1m 55s branch-2.7 passed with JDK v1.7.0_101
        +1 mvninstall 0m 53s the patch passed
        +1 compile 1m 2s the patch passed with JDK v1.8.0_91
        +1 javac 1m 2s the patch passed
        +1 compile 1m 0s the patch passed with JDK v1.7.0_101
        +1 javac 1m 0s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 0m 58s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        -1 whitespace 0m 0s The patch has 2976 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 1m 17s The patch 78 line(s) with tabs.
        +1 findbugs 3m 15s the patch passed
        +1 javadoc 1m 8s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 52s the patch passed with JDK v1.7.0_101
        -1 unit 44m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
        -1 asflicense 0m 20s The patch generated 3 ASF License warnings.
        133m 23s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestLeaseRecovery2
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
        JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.tools.TestJMXGet
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c420dfe
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817775/HDFS-10544-branch-2.7.patch
        JIRA Issue HDFS-10544
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux dd61e512bdd6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / 7c5e743
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16047/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16047/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16047/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16047/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16047/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16047/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 12s branch-2.7 passed +1 compile 1m 1s branch-2.7 passed with JDK v1.8.0_91 +1 compile 1m 2s branch-2.7 passed with JDK v1.7.0_101 +1 checkstyle 0m 24s branch-2.7 passed +1 mvnsite 1m 3s branch-2.7 passed +1 mvneclipse 0m 16s branch-2.7 passed +1 findbugs 3m 8s branch-2.7 passed +1 javadoc 1m 10s branch-2.7 passed with JDK v1.8.0_91 +1 javadoc 1m 55s branch-2.7 passed with JDK v1.7.0_101 +1 mvninstall 0m 53s the patch passed +1 compile 1m 2s the patch passed with JDK v1.8.0_91 +1 javac 1m 2s the patch passed +1 compile 1m 0s the patch passed with JDK v1.7.0_101 +1 javac 1m 0s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2976 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 1m 17s The patch 78 line(s) with tabs. +1 findbugs 3m 15s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 52s the patch passed with JDK v1.7.0_101 -1 unit 44m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_101. -1 asflicense 0m 20s The patch generated 3 ASF License warnings. 133m 23s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.tools.TestJMXGet   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817775/HDFS-10544-branch-2.7.patch JIRA Issue HDFS-10544 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dd61e512bdd6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 7c5e743 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16047/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16047/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16047/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16047/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16047/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16047/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Reported test failures on branch-2.7 patch are unrelated and pass locally. Committing to branch-2.7 soon.

        Show
        zhz Zhe Zhang added a comment - Reported test failures on branch-2.7 patch are unrelated and pass locally. Committing to branch-2.7 soon.
        Hide
        zhz Zhe Zhang added a comment -

        I just pushed to branch-2.7 and branch-2.6. Thanks again for the review from Konstantin Shvachko!

        Show
        zhz Zhe Zhang added a comment - I just pushed to branch-2.7 and branch-2.6. Thanks again for the review from Konstantin Shvachko !
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 16m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 53s branch-2.7 passed
        +1 compile 1m 12s branch-2.7 passed with JDK v1.8.0_91
        +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_101
        +1 checkstyle 0m 26s branch-2.7 passed
        +1 mvnsite 1m 4s branch-2.7 passed
        +1 mvneclipse 0m 18s branch-2.7 passed
        +1 findbugs 3m 5s branch-2.7 passed
        +1 javadoc 1m 15s branch-2.7 passed with JDK v1.8.0_91
        +1 javadoc 1m 51s branch-2.7 passed with JDK v1.7.0_101
        +1 mvninstall 0m 56s the patch passed
        +1 compile 1m 3s the patch passed with JDK v1.8.0_91
        +1 javac 1m 3s the patch passed
        +1 compile 1m 4s the patch passed with JDK v1.7.0_101
        +1 javac 1m 4s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 1m 0s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch has 2976 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 1m 19s The patch 78 line(s) with tabs.
        +1 findbugs 3m 11s the patch passed
        +1 javadoc 1m 7s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 55s the patch passed with JDK v1.7.0_101
        -1 unit 60m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
        -1 asflicense 0m 21s The patch generated 3 ASF License warnings.
        166m 31s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c420dfe
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817775/HDFS-10544-branch-2.7.patch
        JIRA Issue HDFS-10544
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a5d988625ee4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / 771033b
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16049/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16049/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16049/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16049/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16049/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16049/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 16m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 53s branch-2.7 passed +1 compile 1m 12s branch-2.7 passed with JDK v1.8.0_91 +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_101 +1 checkstyle 0m 26s branch-2.7 passed +1 mvnsite 1m 4s branch-2.7 passed +1 mvneclipse 0m 18s branch-2.7 passed +1 findbugs 3m 5s branch-2.7 passed +1 javadoc 1m 15s branch-2.7 passed with JDK v1.8.0_91 +1 javadoc 1m 51s branch-2.7 passed with JDK v1.7.0_101 +1 mvninstall 0m 56s the patch passed +1 compile 1m 3s the patch passed with JDK v1.8.0_91 +1 javac 1m 3s the patch passed +1 compile 1m 4s the patch passed with JDK v1.7.0_101 +1 javac 1m 4s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 2976 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 1m 19s The patch 78 line(s) with tabs. +1 findbugs 3m 11s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 55s the patch passed with JDK v1.7.0_101 -1 unit 60m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_101. -1 asflicense 0m 21s The patch generated 3 ASF License warnings. 166m 31s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817775/HDFS-10544-branch-2.7.patch JIRA Issue HDFS-10544 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a5d988625ee4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 771033b Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16049/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16049/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16049/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16049/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16049/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16049/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.7.3 was already under process, changing the fix-version to 2.7.4.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.7.3 was already under process, changing the fix-version to 2.7.4.

          People

          • Assignee:
            zhz Zhe Zhang
            Reporter:
            zhz Zhe Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development