Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10208

Addendum for HDFS-9579: to handle the case when client machine can't resolve network path

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If DFSClient runs on a machine that can't resolve network path, DNSToSwitchMapping will return DEFAULT_RACK. In addition, if somehow dnsToSwitchMapping.resolve returns null, that will cause exception when it tries to create clientNode. In either case, there is no need to create clientNode and we should treat its network distance with any datanode as Integer.MAX_VALUE.

          clientNode = new NodeBase(clientHostName,
              dnsToSwitchMapping.resolve(nodes).get(0));
      
      1. HDFS-10208.patch
        7 kB
        Ming Ma
      2. HDFS-10208-2.patch
        14 kB
        Ming Ma
      3. HDFS-10208-3.patch
        18 kB
        Ming Ma
      4. HDFS-10208-4.patch
        19 kB
        Ming Ma
      5. HDFS-10208-5.patch
        19 kB
        Ming Ma

        Issue Links

          Activity

          Hide
          mingma Ming Ma added a comment -

          Here is the draft patch. In the case client machine can't resolve network location, the distance will be considered Integer.MAX_VALUE.

          Show
          mingma Ming Ma added a comment - Here is the draft patch. In the case client machine can't resolve network location, the distance will be considered Integer.MAX_VALUE.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 40s trunk passed
          +1 compile 1m 53s trunk passed with JDK v1.8.0_74
          +1 compile 1m 38s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 41s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 3m 50s trunk passed
          +1 javadoc 1m 49s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 29s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 1m 42s the patch passed with JDK v1.8.0_74
          +1 javac 1m 42s the patch passed
          +1 compile 1m 27s the patch passed with JDK v1.7.0_95
          +1 javac 1m 27s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 1m 24s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 8s the patch passed
          +1 javadoc 1m 41s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 16s the patch passed with JDK v1.7.0_95
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74.
          -1 unit 94m 28s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 1m 16s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 83m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 21s Patch generated 1 ASF License warnings.
          220m 57s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795483/HDFS-10208.patch
          JIRA Issue HDFS-10208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0c8984cab230 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fde8ac5
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14946/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14946/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 40s trunk passed +1 compile 1m 53s trunk passed with JDK v1.8.0_74 +1 compile 1m 38s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 41s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 3m 50s trunk passed +1 javadoc 1m 49s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 29s trunk passed with JDK v1.7.0_95 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 1m 42s the patch passed with JDK v1.8.0_74 +1 javac 1m 42s the patch passed +1 compile 1m 27s the patch passed with JDK v1.7.0_95 +1 javac 1m 27s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 8s the patch passed +1 javadoc 1m 41s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 16s the patch passed with JDK v1.7.0_95 +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. -1 unit 94m 28s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 1m 16s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 83m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 21s Patch generated 1 ASF License warnings. 220m 57s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestEditLog   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795483/HDFS-10208.patch JIRA Issue HDFS-10208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0c8984cab230 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fde8ac5 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14946/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14946/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14946/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          asf license will be taken care of by HDFS-10221. The unit test failures aren't related.

          Show
          mingma Ming Ma added a comment - asf license will be taken care of by HDFS-10221 . The unit test failures aren't related.
          Hide
          mingma Ming Ma added a comment -

          The new patch will take care of the case where DNSToSwitchMapping returns NetworkTopology.DEFAULT_RACK, which means it can't resolve network path. In addition, even if the client machine is able to resolve its network location, some applications might not care about these bytes-read-by-distance metrics. Given there is additional cost with this feature such as script launch, extra memory for topology object, the new patch also added a new config to allow applications to enable the feature.

          Show
          mingma Ming Ma added a comment - The new patch will take care of the case where DNSToSwitchMapping returns NetworkTopology.DEFAULT_RACK , which means it can't resolve network path. In addition, even if the client machine is able to resolve its network location, some applications might not care about these bytes-read-by-distance metrics. Given there is additional cost with this feature such as script launch, extra memory for topology object, the new patch also added a new config to allow applications to enable the feature.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 35s trunk passed
          +1 compile 6m 0s trunk passed with JDK v1.8.0_74
          +1 compile 6m 44s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 4s trunk passed
          +1 mvnsite 2m 21s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 5m 14s trunk passed
          +1 javadoc 2m 15s trunk passed with JDK v1.8.0_74
          +1 javadoc 3m 9s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 57s the patch passed
          +1 compile 5m 57s the patch passed with JDK v1.8.0_74
          +1 javac 5m 57s the patch passed
          +1 compile 6m 48s the patch passed with JDK v1.7.0_95
          +1 javac 6m 48s the patch passed
          -1 checkstyle 1m 6s root: patch generated 2 new + 182 unchanged - 0 fixed = 184 total (was 182)
          +1 mvnsite 2m 23s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 6m 2s the patch passed
          +1 javadoc 2m 20s the patch passed with JDK v1.8.0_74
          +1 javadoc 3m 15s the patch passed with JDK v1.7.0_95
          -1 unit 6m 31s hadoop-common in the patch failed with JDK v1.8.0_74.
          +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74.
          -1 unit 58m 31s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          -1 unit 6m 34s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 54m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 26s Patch generated 3 ASF License warnings.
          194m 39s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.TestHFlush
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795901/HDFS-10208-2.patch
          JIRA Issue HDFS-10208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 3aece9ba42e5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f2aec4e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14988/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14988/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 35s trunk passed +1 compile 6m 0s trunk passed with JDK v1.8.0_74 +1 compile 6m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 4s trunk passed +1 mvnsite 2m 21s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 14s trunk passed +1 javadoc 2m 15s trunk passed with JDK v1.8.0_74 +1 javadoc 3m 9s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 57s the patch passed +1 compile 5m 57s the patch passed with JDK v1.8.0_74 +1 javac 5m 57s the patch passed +1 compile 6m 48s the patch passed with JDK v1.7.0_95 +1 javac 6m 48s the patch passed -1 checkstyle 1m 6s root: patch generated 2 new + 182 unchanged - 0 fixed = 184 total (was 182) +1 mvnsite 2m 23s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 6m 2s the patch passed +1 javadoc 2m 20s the patch passed with JDK v1.8.0_74 +1 javadoc 3m 15s the patch passed with JDK v1.7.0_95 -1 unit 6m 31s hadoop-common in the patch failed with JDK v1.8.0_74. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. -1 unit 58m 31s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 6m 34s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 54m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 26s Patch generated 3 ASF License warnings. 194m 39s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.TestHFlush   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795901/HDFS-10208-2.patch JIRA Issue HDFS-10208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 3aece9ba42e5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f2aec4e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14988/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14988/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14988/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          checkstyle isn't an issue. TestHDFSCLI issue has been fixed by HDFS-10228. TestHFlush issue is HDFS-2043. Other failed unit tests passed locally.

          Show
          mingma Ming Ma added a comment - checkstyle isn't an issue. TestHDFSCLI issue has been fixed by HDFS-10228 . TestHFlush issue is HDFS-2043 . Other failed unit tests passed locally.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks for taking care my comment..From now onwards user needs to configure Mapping impl ( and its related values such as script for ScriptBasedMapping) when the client-side network path resolution is enabled. can you mention same in property description.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks for taking care my comment ..From now onwards user needs to configure Mapping impl ( and its related values such as script for ScriptBasedMapping) when the client-side network path resolution is enabled. can you mention same in property description.
          Hide
          mingma Ming Ma added a comment -

          Thanks Brahma Reddy Battula. I have updated the property description in core-default.xml.

          From the investigation of HDFS-10206, the current way of using Topology tree to compute node distance by reference seems too expensive. It means it needs to add nodes to the Topology tree first which could become slow as the tree grows. It also means the tree size could grow unbounded. To solve this issue, we can use network path string comparison instead without Topology tree and the extra HashMap in ClientContext.

          So in summary, the patch has three improvements:

          • Handle the case the client can't resolve network path properly.
          • Make the client-side topology resolution optional.
          • Use string based comparison for network distance calculation.
          Show
          mingma Ming Ma added a comment - Thanks Brahma Reddy Battula . I have updated the property description in core-default.xml. From the investigation of HDFS-10206 , the current way of using Topology tree to compute node distance by reference seems too expensive. It means it needs to add nodes to the Topology tree first which could become slow as the tree grows. It also means the tree size could grow unbounded. To solve this issue, we can use network path string comparison instead without Topology tree and the extra HashMap in ClientContext. So in summary, the patch has three improvements: Handle the case the client can't resolve network path properly. Make the client-side topology resolution optional. Use string based comparison for network distance calculation.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 23s Maven dependency ordering for branch
          +1 mvninstall 7m 20s trunk passed
          +1 compile 6m 42s trunk passed with JDK v1.8.0_77
          +1 compile 7m 16s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 2m 30s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 5m 38s trunk passed
          +1 javadoc 2m 29s trunk passed with JDK v1.8.0_77
          +1 javadoc 3m 19s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 7s the patch passed
          +1 compile 6m 43s the patch passed with JDK v1.8.0_77
          +1 javac 6m 43s the patch passed
          +1 compile 7m 37s the patch passed with JDK v1.7.0_95
          +1 javac 7m 37s the patch passed
          -1 checkstyle 1m 10s root: patch generated 5 new + 278 unchanged - 0 fixed = 283 total (was 278)
          +1 mvnsite 2m 30s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 6m 17s the patch passed
          +1 javadoc 2m 34s the patch passed with JDK v1.8.0_77
          +1 javadoc 3m 18s the patch passed with JDK v1.7.0_95
          -1 unit 7m 23s hadoop-common in the patch failed with JDK v1.8.0_77.
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          -1 unit 59m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 7m 15s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 52m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          201m 57s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.net.TestDNS
            hadoop.hdfs.TestHFlush
          JDK v1.7.0_95 Failed junit tests hadoop.net.TestClusterTopology
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.shortcircuit.TestShortCircuitCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796550/HDFS-10208-3.patch
          JIRA Issue HDFS-10208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux aaeb53d893b1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f65f5b1
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15061/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15061/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 7m 20s trunk passed +1 compile 6m 42s trunk passed with JDK v1.8.0_77 +1 compile 7m 16s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 10s trunk passed +1 mvnsite 2m 30s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 5m 38s trunk passed +1 javadoc 2m 29s trunk passed with JDK v1.8.0_77 +1 javadoc 3m 19s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 7s the patch passed +1 compile 6m 43s the patch passed with JDK v1.8.0_77 +1 javac 6m 43s the patch passed +1 compile 7m 37s the patch passed with JDK v1.7.0_95 +1 javac 7m 37s the patch passed -1 checkstyle 1m 10s root: patch generated 5 new + 278 unchanged - 0 fixed = 283 total (was 278) +1 mvnsite 2m 30s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 6m 17s the patch passed +1 javadoc 2m 34s the patch passed with JDK v1.8.0_77 +1 javadoc 3m 18s the patch passed with JDK v1.7.0_95 -1 unit 7m 23s hadoop-common in the patch failed with JDK v1.8.0_77. +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. -1 unit 59m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 7m 15s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 52m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 201m 57s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.TestHFlush JDK v1.7.0_95 Failed junit tests hadoop.net.TestClusterTopology   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestHFlush   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796550/HDFS-10208-3.patch JIRA Issue HDFS-10208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux aaeb53d893b1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f65f5b1 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15061/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15061/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Updated patch to address checkstyle issue. Failed tests aren't related.

          Show
          mingma Ming Ma added a comment - Updated patch to address checkstyle issue. Failed tests aren't related.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 55s trunk passed
          +1 compile 6m 5s trunk passed with JDK v1.8.0_77
          +1 compile 6m 57s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 2m 24s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 5m 3s trunk passed
          +1 javadoc 2m 24s trunk passed with JDK v1.8.0_77
          +1 javadoc 3m 17s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 59s the patch passed
          +1 compile 5m 49s the patch passed with JDK v1.8.0_77
          +1 javac 5m 49s the patch passed
          +1 compile 6m 45s the patch passed with JDK v1.7.0_95
          +1 javac 6m 45s the patch passed
          -1 checkstyle 1m 6s root: patch generated 2 new + 278 unchanged - 0 fixed = 280 total (was 278)
          +1 mvnsite 2m 25s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 5m 52s the patch passed
          +1 javadoc 2m 29s the patch passed with JDK v1.8.0_77
          +1 javadoc 3m 23s the patch passed with JDK v1.7.0_95
          +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          -1 unit 75m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          +1 unit 7m 38s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 69m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          230m 27s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797774/HDFS-10208-4.patch
          JIRA Issue HDFS-10208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 27eac7335189 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9cb0c96
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15122/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15122/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 6m 5s trunk passed with JDK v1.8.0_77 +1 compile 6m 57s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 2m 24s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 5m 3s trunk passed +1 javadoc 2m 24s trunk passed with JDK v1.8.0_77 +1 javadoc 3m 17s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 59s the patch passed +1 compile 5m 49s the patch passed with JDK v1.8.0_77 +1 javac 5m 49s the patch passed +1 compile 6m 45s the patch passed with JDK v1.7.0_95 +1 javac 6m 45s the patch passed -1 checkstyle 1m 6s root: patch generated 2 new + 278 unchanged - 0 fixed = 280 total (was 278) +1 mvnsite 2m 25s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 52s the patch passed +1 javadoc 2m 29s the patch passed with JDK v1.8.0_77 +1 javadoc 3m 23s the patch passed with JDK v1.7.0_95 +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.8.0_77. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. -1 unit 75m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_77. +1 unit 7m 38s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 69m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 230m 27s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797774/HDFS-10208-4.patch JIRA Issue HDFS-10208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 27eac7335189 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9cb0c96 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15122/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15122/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15122/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks Ming Ma for the patch!

          The default value for the new config is false, meaning we want to disable the feature for HDFS-9579 by default. Is that desirable? I would think that we should be able to enable it by default unless people want to turn it off. I'd like to hear your thought on the default...

          I see that the lookup map in ClientContext was removed as part of this patch. However, if getNetworkDistance() is called repeatedly on the same datanodes, there is definitely a benefit in caching the result of the computation. Is the concern for removing the nodeToDistance map about its size?

          Show
          sjlee0 Sangjin Lee added a comment - Thanks Ming Ma for the patch! The default value for the new config is false, meaning we want to disable the feature for HDFS-9579 by default. Is that desirable? I would think that we should be able to enable it by default unless people want to turn it off. I'd like to hear your thought on the default... I see that the lookup map in ClientContext was removed as part of this patch. However, if getNetworkDistance() is called repeatedly on the same datanodes, there is definitely a benefit in caching the result of the computation. Is the concern for removing the nodeToDistance map about its size?
          Hide
          mingma Ming Ma added a comment -

          Thanks Sangjin Lee!

          • By default ScriptBasedMapping is used but without any script file defined. That means by default DEFAULT_RACK will be returned and thus cause invalid results as Brahma Reddy Battula pointed out in comment. In addition, there is a cost for topology lookup as discussed in HDFS-10203. So for applications that don't care about it, there is no perf regression if it is turned off by default.
          • There is some complexity in cache management.
            • The cache size is a function of the max number of DNs across clusters, thus it will take up some memory on the client process although relatively small.
            • The cache freshness for long running service(although it isn't the immediate scenario we want to address). The cache is based on ip address, it is possible a given ip address changes its network location since the client process caches its network location.
            • The targeted scenario is MR task which is short duration which access small number of blocks and the cache becomes less useful.
            • The perf without cache might not be an issue. The frequency of getNetworkDistance is one per HDFS block read so it shouldn't be high assuming large block size. Also getNetworkDistance doesn't involve topology lookup so it should still be fast.

          What do you think?

          Show
          mingma Ming Ma added a comment - Thanks Sangjin Lee ! By default ScriptBasedMapping is used but without any script file defined. That means by default DEFAULT_RACK will be returned and thus cause invalid results as Brahma Reddy Battula pointed out in comment . In addition, there is a cost for topology lookup as discussed in HDFS-10203 . So for applications that don't care about it, there is no perf regression if it is turned off by default. There is some complexity in cache management. The cache size is a function of the max number of DNs across clusters, thus it will take up some memory on the client process although relatively small. The cache freshness for long running service(although it isn't the immediate scenario we want to address). The cache is based on ip address, it is possible a given ip address changes its network location since the client process caches its network location. The targeted scenario is MR task which is short duration which access small number of blocks and the cache becomes less useful. The perf without cache might not be an issue. The frequency of getNetworkDistance is one per HDFS block read so it shouldn't be high assuming large block size. Also getNetworkDistance doesn't involve topology lookup so it should still be fast. What do you think?
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the clarification. I am then comfortable with those choices.

          It does appear the checkstyle violations are fixable. Could you please fix them? With that, it LGTM, but I'd like to give time for Brahma Reddy Battula to take a look at it too.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the clarification. I am then comfortable with those choices. It does appear the checkstyle violations are fixable. Could you please fix them? With that, it LGTM, but I'd like to give time for Brahma Reddy Battula to take a look at it too.
          Hide
          mingma Ming Ma added a comment -

          Thanks. Sure I can fix the checkstyle issue, although this specific issue is applicable for most existing properties defined in CommonConfigurationKeysPublic.

          Show
          mingma Ming Ma added a comment - Thanks. Sure I can fix the checkstyle issue, although this specific issue is applicable for most existing properties defined in CommonConfigurationKeysPublic.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          LGTM,Sangjin Lee thanks for pinging me..

          applicable for most existing properties defined in CommonConfigurationKeysPublic.

          yes..May be , To make Jenkins happy can update patch with checkstyle fixes..

          Show
          brahmareddy Brahma Reddy Battula added a comment - LGTM, Sangjin Lee thanks for pinging me.. applicable for most existing properties defined in CommonConfigurationKeysPublic. yes..May be , To make Jenkins happy can update patch with checkstyle fixes..
          Hide
          mingma Ming Ma added a comment -

          Here is the updated patch. Thank you Sangjin Lee and Brahma Reddy Battula.

          Show
          mingma Ming Ma added a comment - Here is the updated patch. Thank you Sangjin Lee and Brahma Reddy Battula .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 6m 51s trunk passed
          +1 compile 5m 55s trunk passed with JDK v1.8.0_77
          +1 compile 6m 47s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 7s trunk passed
          +1 mvnsite 2m 28s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 5m 7s trunk passed
          +1 javadoc 2m 21s trunk passed with JDK v1.8.0_77
          +1 javadoc 3m 15s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 58s the patch passed
          +1 compile 6m 1s the patch passed with JDK v1.8.0_77
          +1 javac 6m 1s the patch passed
          +1 compile 6m 57s the patch passed with JDK v1.7.0_95
          +1 javac 6m 57s the patch passed
          +1 checkstyle 1m 8s the patch passed
          +1 mvnsite 2m 26s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 5m 55s the patch passed
          +1 javadoc 2m 24s the patch passed with JDK v1.8.0_77
          +1 javadoc 3m 22s the patch passed with JDK v1.7.0_95
          -1 unit 7m 55s hadoop-common in the patch failed with JDK v1.8.0_77.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          -1 unit 90m 8s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 8m 30s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 99m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 50s Patch does not generate ASF License warnings.
          276m 39s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.datanode.TestDataNodeLifeline
            hadoop.hdfs.TestFileChecksum
            hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestMultiThreadedHflush
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestParallelShortCircuitRead
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
          JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.TestFileChecksum
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
            org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798543/HDFS-10208-5.patch
          JIRA Issue HDFS-10208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 734412987f59 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e0cb426
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15152/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15152/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 5m 55s trunk passed with JDK v1.8.0_77 +1 compile 6m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 7s trunk passed +1 mvnsite 2m 28s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 7s trunk passed +1 javadoc 2m 21s trunk passed with JDK v1.8.0_77 +1 javadoc 3m 15s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 58s the patch passed +1 compile 6m 1s the patch passed with JDK v1.8.0_77 +1 javac 6m 1s the patch passed +1 compile 6m 57s the patch passed with JDK v1.7.0_95 +1 javac 6m 57s the patch passed +1 checkstyle 1m 8s the patch passed +1 mvnsite 2m 26s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 55s the patch passed +1 javadoc 2m 24s the patch passed with JDK v1.8.0_77 +1 javadoc 3m 22s the patch passed with JDK v1.7.0_95 -1 unit 7m 55s hadoop-common in the patch failed with JDK v1.8.0_77. +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. -1 unit 90m 8s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 8m 30s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 99m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 50s Patch does not generate ASF License warnings. 276m 39s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.TestFileChecksum   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestParallelShortCircuitRead JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestFileChecksum   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.shortcircuit.TestShortCircuitCache JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798543/HDFS-10208-5.patch JIRA Issue HDFS-10208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 734412987f59 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0cb426 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15152/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15152/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15152/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          TestReadStripedFileWithDecoding and TestWriteReadStripedFile aren't related. All other tests passed locally.

          Show
          mingma Ming Ma added a comment - TestReadStripedFileWithDecoding and TestWriteReadStripedFile aren't related. All other tests passed locally.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Sorry for the delay Ming Ma. I am +1. Unless I hear additional comments (cc Brahma Reddy Battula), I'll commit it this evening. Let me know which versions this should be committed to other than trunk and branch-2 (2.9.0).

          Show
          sjlee0 Sangjin Lee added a comment - Sorry for the delay Ming Ma . I am +1. Unless I hear additional comments (cc Brahma Reddy Battula ), I'll commit it this evening. Let me know which versions this should be committed to other than trunk and branch-2 (2.9.0).
          Hide
          sjlee0 Sangjin Lee added a comment -

          Committed it to trunk and branch-2. I did notice an existing unit test failure on branch-2, for which I filed HDFS-10415.

          Thanks Ming Ma for your contribution, and Brahma Reddy Battula for your review!

          Show
          sjlee0 Sangjin Lee added a comment - Committed it to trunk and branch-2. I did notice an existing unit test failure on branch-2, for which I filed HDFS-10415 . Thanks Ming Ma for your contribution, and Brahma Reddy Battula for your review!
          Hide
          mingma Ming Ma added a comment -
          Show
          mingma Ming Ma added a comment - Thank you Sangjin Lee and Brahma Reddy Battula !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9773 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9773/)
          HDFS-10208. Addendum for HDFS-9579: to handle the case when client (sjlee: rev 61f46be071e42f9eb49a54b1bd2e54feac59f808)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NodeBase.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9773 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9773/ ) HDFS-10208 . Addendum for HDFS-9579 : to handle the case when client (sjlee: rev 61f46be071e42f9eb49a54b1bd2e54feac59f808) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NodeBase.java

            People

            • Assignee:
              mingma Ming Ma
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development