Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9724

Degraded performance in WebHDFS listing as it does not reuse ObjectMapper

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: performance
    • Labels:
      None
    • Target Version/s:

      Description

      Eremikhin Alexey reported at user ML that WebHDFS listing is too slow because of HDFS-6565.
      https://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201601.mbox/%3C56AB857B.6020105%40corp.badoo.com%3E

      1. HDFS-9724.01.patch
        3 kB
        Akira Ajisaka
      2. HDFS-9724-branch-2.7.01.patch
        4 kB
        Akira Ajisaka
      3. HDFS-9724.02.patch
        4 kB
        Akira Ajisaka
      4. HDFS-9724.03.patch
        5 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks everyone.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks everyone.
          Hide
          wheat9 Haohui Mai added a comment - - edited

          I've committed to trunk, branch-2, branch-2.8 and branch-2.7. Thanks the contribution, Akira Ajisaka.

          Show
          wheat9 Haohui Mai added a comment - - edited I've committed to trunk, branch-2, branch-2.8 and branch-2.7. Thanks the contribution, Akira Ajisaka .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9246 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9246/)
          HDFS-9724. Degraded performance in WebHDFS listing as it does not reuse (haohui: rev 1bcfab8e7fd8562f1829ac484d2f6c91f7afe3d6)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9246 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9246/ ) HDFS-9724 . Degraded performance in WebHDFS listing as it does not reuse (haohui: rev 1bcfab8e7fd8562f1829ac484d2f6c91f7afe3d6) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 from me too. Thanks, everyone!

          Show
          cnauroth Chris Nauroth added a comment - +1 from me too. Thanks, everyone!
          Hide
          wheat9 Haohui Mai added a comment -

          +1. I'll commit it shortly.

          Show
          wheat9 Haohui Mai added a comment - +1. I'll commit it shortly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 22s Maven dependency ordering for branch
          +1 mvninstall 9m 1s trunk passed
          +1 compile 2m 14s trunk passed with JDK v1.8.0_66
          +1 compile 2m 1s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 2m 3s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 4m 40s trunk passed
          +1 javadoc 2m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 3s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 1m 53s the patch passed
          +1 compile 2m 24s the patch passed with JDK v1.8.0_66
          +1 javac 2m 24s the patch passed
          +1 compile 2m 2s the patch passed with JDK v1.7.0_91
          +1 javac 2m 2s the patch passed
          +1 checkstyle 0m 34s the patch passed
          +1 mvnsite 2m 2s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 10s the patch passed
          +1 javadoc 1m 35s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 16s the patch passed with JDK v1.7.0_91
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 85m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 83m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          218m 26s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786227/HDFS-9724.03.patch
          JIRA Issue HDFS-9724
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 03a84adeca2e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 22a2b22
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14381/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14381/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 22s Maven dependency ordering for branch +1 mvninstall 9m 1s trunk passed +1 compile 2m 14s trunk passed with JDK v1.8.0_66 +1 compile 2m 1s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 36s trunk passed +1 mvnsite 2m 3s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 4m 40s trunk passed +1 javadoc 2m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 3s trunk passed with JDK v1.7.0_91 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 2m 24s the patch passed with JDK v1.8.0_66 +1 javac 2m 24s the patch passed +1 compile 2m 2s the patch passed with JDK v1.7.0_91 +1 javac 2m 2s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 2m 2s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 10s the patch passed +1 javadoc 1m 35s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 16s the patch passed with JDK v1.7.0_91 +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 85m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 83m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 218m 26s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.tracing.TestTracing   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786227/HDFS-9724.03.patch JIRA Issue HDFS-9724 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 03a84adeca2e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22a2b22 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14381/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14381/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Haohui Mai for the review.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Haohui Mai for the review.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          03 patch

          • Commented why we can and should reuse ObjectMapper instance.
          Show
          ajisakaa Akira Ajisaka added a comment - 03 patch Commented why we can and should reuse ObjectMapper instance.
          Hide
          wheat9 Haohui Mai added a comment -

          AFAIK, we don't have a re-entrant call pattern in WebHDFS though, and we just need to worry about thread-safety. Am I missing something?

          Thanks Chris for bringing this up. I just checked the code and confirmed your statement. I think this change is safe.

          It might be beneficial to add a comment to document the assumption.

          Show
          wheat9 Haohui Mai added a comment - AFAIK, we don't have a re-entrant call pattern in WebHDFS though, and we just need to worry about thread-safety. Am I missing something? Thanks Chris for bringing this up. I just checked the code and confirmed your statement. I think this change is safe. It might be beneficial to add a comment to document the assumption.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 6m 50s trunk passed
          +1 compile 1m 21s trunk passed with JDK v1.8.0_66
          +1 compile 1m 21s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 29s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 51s trunk passed
          +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 7s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 1m 14s the patch passed with JDK v1.8.0_66
          +1 javac 1m 14s the patch passed
          +1 compile 1m 19s the patch passed with JDK v1.7.0_91
          +1 javac 1m 19s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 12s the patch passed
          +1 javadoc 1m 20s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 7s the patch passed with JDK v1.7.0_91
          +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 53m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          +1 unit 50m 16s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          142m 4s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestReconstructStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785919/HDFS-9724.02.patch
          JIRA Issue HDFS-9724
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2f8379307fbb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0437575
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14354/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14354/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14354/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14354/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 6m 50s trunk passed +1 compile 1m 21s trunk passed with JDK v1.8.0_66 +1 compile 1m 21s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 29s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 51s trunk passed +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 7s trunk passed with JDK v1.7.0_91 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 14s the patch passed with JDK v1.8.0_66 +1 javac 1m 14s the patch passed +1 compile 1m 19s the patch passed with JDK v1.7.0_91 +1 javac 1m 19s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 12s the patch passed +1 javadoc 1m 20s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 7s the patch passed with JDK v1.7.0_91 +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 53m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 unit 50m 16s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 142m 4s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestReconstructStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785919/HDFS-9724.02.patch JIRA Issue HDFS-9724 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2f8379307fbb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0437575 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14354/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14354/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14354/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14354/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Should I create jira to deal with it or a better way?

          You can create a jira. Thanks Yiqun Lin.

          Show
          ajisakaa Akira Ajisaka added a comment - Should I create jira to deal with it or a better way? You can create a jira. Thanks Yiqun Lin .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Chris Nauroth for the detailed comment.

          AFAIK, we don't have a re-entrant call pattern in WebHDFS though, and we just need to worry about thread-safety.

          Agree.

          In addition to the changes in the current patch, I also see one more new ObjectMapper() call inside WebHdfsFileSystem.

          Nice catch! I'll fix it shortly.

          We probably need a comprehensive review across the whole codebase to look for this pattern, but we can keep the current JIRA's scope focused on WebHDFS, since we know it is causing a performance problem there.

          Agree. I'd like to keep the current scope.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Chris Nauroth for the detailed comment. AFAIK, we don't have a re-entrant call pattern in WebHDFS though, and we just need to worry about thread-safety. Agree. In addition to the changes in the current patch, I also see one more new ObjectMapper() call inside WebHdfsFileSystem. Nice catch! I'll fix it shortly. We probably need a comprehensive review across the whole codebase to look for this pattern, but we can keep the current JIRA's scope focused on WebHDFS, since we know it is causing a performance problem there. Agree. I'd like to keep the current scope.
          Hide
          linyiqun Yiqun Lin added a comment -

          There are many other places to call ObjectMapper like this way in Yarn and MapReduce. Maybe we could change them to the single-instance way and reducing the instance created. Should I create jira to deal with it or a better way?

          Show
          linyiqun Yiqun Lin added a comment - There are many other places to call ObjectMapper like this way in Yarn and MapReduce. Maybe we could change them to the single-instance way and reducing the instance created. Should I create jira to deal with it or a better way?
          Hide
          cnauroth Chris Nauroth added a comment -

          ...yes, as long as you always configure instance before use...

          Most of the ObjectMapper API is thread-safe after initialization. That means that if we want to call methods like ObjectMapper#enable and ObjectMapper#disable to control how the serialization/deserialization works, then we'd need to do it during some single-threaded initialization phase, such as process startup. After that, multiple threads may call the serialization/deserialization methods concurrently. I've used Jackson successfully this way in other multi-threaded codebases.

          I'm thinking the method is not reentrant. ObjectMapper uses thread local buffer for parsers and generators to achieve thread-safety.

          I'm curious why the question was brought up as re-entrancy instead of thread-safety. If we have a re-entrant call pattern, where the same thread ends up with more than one stack frame inside the serialization/deserialization methods, then the later call would stomp on the earlier call's usage of thread-local storage. AFAIK, we don't have a re-entrant call pattern in WebHDFS though, and we just need to worry about thread-safety. Am I missing something?

          In addition to the changes in the current patch, I also see one more new ObjectMapper() call inside WebHdfsFileSystem. We probably need a comprehensive review across the whole codebase to look for this pattern, but we can keep the current JIRA's scope focused on WebHDFS, since we know it is causing a performance problem there.

          Show
          cnauroth Chris Nauroth added a comment - ...yes, as long as you always configure instance before use... Most of the ObjectMapper API is thread-safe after initialization. That means that if we want to call methods like ObjectMapper#enable and ObjectMapper#disable to control how the serialization/deserialization works, then we'd need to do it during some single-threaded initialization phase, such as process startup. After that, multiple threads may call the serialization/deserialization methods concurrently. I've used Jackson successfully this way in other multi-threaded codebases. I'm thinking the method is not reentrant. ObjectMapper uses thread local buffer for parsers and generators to achieve thread-safety. I'm curious why the question was brought up as re-entrancy instead of thread-safety. If we have a re-entrant call pattern, where the same thread ends up with more than one stack frame inside the serialization/deserialization methods, then the later call would stomp on the earlier call's usage of thread-local storage. AFAIK, we don't have a re-entrant call pattern in WebHDFS though, and we just need to worry about thread-safety. Am I missing something? In addition to the changes in the current patch, I also see one more new ObjectMapper() call inside WebHdfsFileSystem . We probably need a comprehensive review across the whole codebase to look for this pattern, but we can keep the current JIRA's scope focused on WebHDFS, since we know it is causing a performance problem there.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Is ObjectMapper#writeValueAsString() reentrant?

          I'm thinking the method is not reentrant. ObjectMapper uses thread local buffer for parsers and generators to achieve thread-safety.

          http://wiki.fasterxml.com/JacksonFAQ

          Is ObjectMapper thread-safe?
          Short answer: yes
          Long answer: yes, as long as you always configure instance before use, and do not call configure methods during operation (or synchronize such calls appropriately). Usually it is better to construct separate mapper instance if configurations differ in any case.

          Now the wiki is in trouble. Cache: http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp

          Show
          ajisakaa Akira Ajisaka added a comment - Is ObjectMapper#writeValueAsString() reentrant? I'm thinking the method is not reentrant. ObjectMapper uses thread local buffer for parsers and generators to achieve thread-safety. http://wiki.fasterxml.com/JacksonFAQ Is ObjectMapper thread-safe? Short answer: yes Long answer: yes, as long as you always configure instance before use, and do not call configure methods during operation (or synchronize such calls appropriately). Usually it is better to construct separate mapper instance if configurations differ in any case. Now the wiki is in trouble. Cache: http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp
          Hide
          wheat9 Haohui Mai added a comment -

          Is

          {ObjectMapper#writeValueAsString()}

          reentrant?

          Show
          wheat9 Haohui Mai added a comment - Is {ObjectMapper#writeValueAsString()} reentrant?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Haohui Mai, could you review the patch?

          Show
          ajisakaa Akira Ajisaka added a comment - Haohui Mai , could you review the patch?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Alexey Eremikhin for reporting this issue and verifying the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Alexey Eremikhin for reporting this issue and verifying the patch.
          Hide
          alexxz Alexey Eremikhin added a comment -

          I can confirm suggested patch solves my original issue

          alexxz@hadoop1.mlan:~> time curl  'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc 
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
          100 2469k    0 2469k    0     0  14.5M      0 --:--:-- --:--:-- --:--:-- 14.9M
            10002   10002 2528929
          
          real	0m0.180s
          user	0m0.096s
          sys	0m0.012s
          

          Thank you very much!

          Show
          alexxz Alexey Eremikhin added a comment - I can confirm suggested patch solves my original issue alexxz@hadoop1.mlan:~> time curl 'http: //hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2469k 0 2469k 0 0 14.5M 0 --:--:-- --:--:-- --:--:-- 14.9M 10002 10002 2528929 real 0m0.180s user 0m0.096s sys 0m0.012s Thank you very much!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 19s trunk passed
          +1 compile 0m 45s trunk passed with JDK v1.8.0_66
          +1 compile 0m 44s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 3s trunk passed
          +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 56s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.8.0_66
          +1 javac 0m 43s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.7.0_91
          +1 javac 0m 47s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 16s the patch passed
          +1 javadoc 1m 7s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91
          -1 unit 58m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 52m 54s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          138m 49s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785317/HDFS-9724.01.patch
          JIRA Issue HDFS-9724
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4cff34b2f0f0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 772ea7b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14298/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14298/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 19s trunk passed +1 compile 0m 45s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 3s trunk passed +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 56s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 49s the patch passed +1 compile 0m 43s the patch passed with JDK v1.8.0_66 +1 javac 0m 43s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_91 +1 javac 0m 47s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91 -1 unit 58m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 52m 54s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 138m 49s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785317/HDFS-9724.01.patch JIRA Issue HDFS-9724 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4cff34b2f0f0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 772ea7b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14298/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14298/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          -1 mvninstall 0m 26s root in branch-2.7 failed.
          -1 compile 0m 8s hadoop-hdfs in branch-2.7 failed with JDK v1.8.0_72.
          -1 compile 0m 7s hadoop-hdfs in branch-2.7 failed with JDK v1.7.0_91.
          +1 checkstyle 0m 7s branch-2.7 passed
          -1 mvnsite 0m 7s hadoop-hdfs in branch-2.7 failed.
          -1 mvneclipse 0m 7s hadoop-hdfs in branch-2.7 failed.
          -1 findbugs 0m 7s hadoop-hdfs in branch-2.7 failed.
          -1 javadoc 0m 5s hadoop-hdfs in branch-2.7 failed with JDK v1.8.0_72.
          -1 javadoc 0m 7s hadoop-hdfs in branch-2.7 failed with JDK v1.7.0_91.
          -1 mvninstall 0m 6s hadoop-hdfs in the patch failed.
          -1 compile 0m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 javac 0m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 compile 0m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 javac 0m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 checkstyle 0m 7s the patch passed
          -1 mvnsite 0m 7s hadoop-hdfs in the patch failed.
          -1 mvneclipse 0m 8s hadoop-hdfs in the patch failed.
          -1 whitespace 0m 1s The patch has 493 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 12s The patch has 57 line(s) with tabs.
          -1 findbugs 0m 7s hadoop-hdfs in the patch failed.
          -1 javadoc 0m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 javadoc 0m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 unit 0m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 0m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 43m 52s Patch generated 57 ASF License warnings.
          59m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2016-01-30
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785318/HDFS-9724-branch-2.7.01.patch
          JIRA Issue HDFS-9724
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 23c737004da1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / ff04a18
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-mvninstall-root.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-mvneclipse-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-mvneclipse-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14299/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 46MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14299/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 mvninstall 0m 26s root in branch-2.7 failed. -1 compile 0m 8s hadoop-hdfs in branch-2.7 failed with JDK v1.8.0_72. -1 compile 0m 7s hadoop-hdfs in branch-2.7 failed with JDK v1.7.0_91. +1 checkstyle 0m 7s branch-2.7 passed -1 mvnsite 0m 7s hadoop-hdfs in branch-2.7 failed. -1 mvneclipse 0m 7s hadoop-hdfs in branch-2.7 failed. -1 findbugs 0m 7s hadoop-hdfs in branch-2.7 failed. -1 javadoc 0m 5s hadoop-hdfs in branch-2.7 failed with JDK v1.8.0_72. -1 javadoc 0m 7s hadoop-hdfs in branch-2.7 failed with JDK v1.7.0_91. -1 mvninstall 0m 6s hadoop-hdfs in the patch failed. -1 compile 0m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 javac 0m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 compile 0m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 javac 0m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 checkstyle 0m 7s the patch passed -1 mvnsite 0m 7s hadoop-hdfs in the patch failed. -1 mvneclipse 0m 8s hadoop-hdfs in the patch failed. -1 whitespace 0m 1s The patch has 493 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 12s The patch has 57 line(s) with tabs. -1 findbugs 0m 7s hadoop-hdfs in the patch failed. -1 javadoc 0m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 javadoc 0m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 unit 0m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 0m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 43m 52s Patch generated 57 ASF License warnings. 59m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:date2016-01-30 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785318/HDFS-9724-branch-2.7.01.patch JIRA Issue HDFS-9724 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 23c737004da1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / ff04a18 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt mvneclipse https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-mvneclipse-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt mvneclipse https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-mvneclipse-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14299/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14299/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 46MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14299/console This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for working on this, Akira Ajisaka. Patch looks good to me. +1 (non-binding)

          As we don't call any configuration methods after created, we can and should reuse ObjectMapper objects as much as possible for performance reasons. ObjectMapper saves a lot of information about the Java classes that they are mapping.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for working on this, Akira Ajisaka . Patch looks good to me. +1 (non-binding) As we don't call any configuration methods after created, we can and should reuse ObjectMapper objects as much as possible for performance reasons. ObjectMapper saves a lot of information about the Java classes that they are mapping.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Created a patch for branch-2.7.

          Show
          ajisakaa Akira Ajisaka added a comment - Created a patch for branch-2.7.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch to re-use ObjectMapper instance. After applying the patch, WebHDFS listing gets faster.

          [root@trunk ~]# time curl 'http://trunk:50070/webhdfs/v1/tmp/test/?op=LISTSTATUS' | wc
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
          100 2440k    0 2440k    0     0  18.6M      0 --:--:-- --:--:-- --:--:-- 18.6M
            10002   10002 2498929
          
          real	0m0.136s
          user	0m0.054s
          sys	0m0.011s
          
          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch to re-use ObjectMapper instance. After applying the patch, WebHDFS listing gets faster. [root@trunk ~]# time curl 'http://trunk:50070/webhdfs/v1/tmp/test/?op=LISTSTATUS' | wc % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2440k 0 2440k 0 0 18.6M 0 --:--:-- --:--:-- --:--:-- 18.6M 10002 10002 2498929 real 0m0.136s user 0m0.054s sys 0m0.011s
          Hide
          ajisakaa Akira Ajisaka added a comment -

          My test on trunk. Created 10000 files at /tmp/test and list them.

          [root@trunk ~]# time curl 'http://trunk:50070/webhdfs/v1/tmp/test/?op=LISTSTATUS' | wc
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
          100 2440k    0 2440k    0     0   408k      0 --:--:--  0:00:05 --:--:--  413k
            10002   10002 2498929
          
          real	0m5.978s
          user	0m0.093s
          sys	0m0.011s
          
          Show
          ajisakaa Akira Ajisaka added a comment - My test on trunk. Created 10000 files at /tmp/test and list them. [root@trunk ~]# time curl 'http://trunk:50070/webhdfs/v1/tmp/test/?op=LISTSTATUS' | wc % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2440k 0 2440k 0 0 408k 0 --:--:-- 0:00:05 --:--:-- 413k 10002 10002 2498929 real 0m5.978s user 0m0.093s sys 0m0.011s
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Eremikhin's benchmark:

          • Hadoop 2.7.1
            alexxz@hadoop1.mlan:~> time curl  'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
              % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                             Dload  Upload   Total   Spent    Left  Speed
            100 2469k    0 2469k    0     0   335k      0 --:--:--  0:00:07 --:--:--  338k
              10002   10002 2528929
            
            real    0m7.397s
            user    0m0.084s
            sys    0m0.028s
            
          • Hadoop 2.7.1 with the patch which reverts HDFS-6565
            alexxz@hadoop1.mlan:~> time curl  'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
              % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                             Dload  Upload   Total   Spent    Left  Speed
            100 2469k    0 2469k    0     0  14.5M      0 --:--:-- --:--:-- --:--:-- 15.0M
              10002   10002 2528929
            
            real    0m0.180s
            user    0m0.100s
            sys    0m0.012s
            
          Show
          ajisakaa Akira Ajisaka added a comment - Eremikhin's benchmark: Hadoop 2.7.1 alexxz@hadoop1.mlan:~> time curl 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2469k 0 2469k 0 0 335k 0 --:--:-- 0:00:07 --:--:-- 338k 10002 10002 2528929 real 0m7.397s user 0m0.084s sys 0m0.028s Hadoop 2.7.1 with the patch which reverts HDFS-6565 alexxz@hadoop1.mlan:~> time curl 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2469k 0 2469k 0 0 14.5M 0 --:--:-- --:--:-- --:--:-- 15.0M 10002 10002 2528929 real 0m0.180s user 0m0.100s sys 0m0.012s

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              ajisakaa Akira Ajisaka
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development