Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11943

[Erasure coding] Warn log frequently print to screen in doEncode/doDecode functions

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: erasure-coding, native
    • Labels:
      None
    • Environment:

      cluster: 3 nodes
      os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, Ubuntu4.4.0-31-generic)
      hadoop version: hadoop-3.0.0-alpha4
      erasure coding: XOR-2-1-64k and enabled Intel ISA-L
      hadoop fs -put file /

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Flags:
      Patch

      Description

      when i write file to hdfs on above environment, the hdfs client frequently print warn log of use direct ByteBuffer inputs/outputs in doEncode function to screen, detail information as follows:
      2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: convertToByteBufferState is invoked, not efficiently. Please use direct ByteBuffer inputs/outputs

      1. HDFS-11943.002.patch
        1 kB
        liaoyuxiangqin
      2. HDFS-11943.003.patch
        2 kB
        liaoyuxiangqin
      3. HDFS-11943.004.patch
        3 kB
        liaoyuxiangqin
      4. HDFS-11943.005.patch
        3 kB
        liaoyuxiangqin
      5. HDFS-11943.patch
        1 kB
        liaoyuxiangqin

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11922 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11922/)
        HDFS-11943. [Erasure coding] Warn log frequently print to screen in (kai.zheng: rev 379f19a2c768ac3cf668ad28ce6419ca56a01b07)

        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractNativeRawEncoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractNativeRawDecoder.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11922 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11922/ ) HDFS-11943 . [Erasure coding] Warn log frequently print to screen in (kai.zheng: rev 379f19a2c768ac3cf668ad28ce6419ca56a01b07) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractNativeRawEncoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractNativeRawDecoder.java
        Hide
        drankye Kai Zheng added a comment -

        This shouldn't forget Andrew Wang's suggestion and SammiChen's review. Thanks!

        Show
        drankye Kai Zheng added a comment - This shouldn't forget Andrew Wang 's suggestion and SammiChen 's review. Thanks!
        Hide
        drankye Kai Zheng added a comment -

        Committed to trunk. Thanks liaoyuxiangqin for reporting this and also providing the patch!

        Show
        drankye Kai Zheng added a comment - Committed to trunk. Thanks liaoyuxiangqin for reporting this and also providing the patch!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 17m 41s trunk passed
        +1 compile 19m 4s trunk passed
        +1 checkstyle 0m 53s trunk passed
        +1 mvnsite 1m 35s trunk passed
        +1 findbugs 1m 38s trunk passed
        +1 javadoc 0m 53s trunk passed
        +1 mvninstall 0m 53s the patch passed
        +1 compile 11m 22s the patch passed
        +1 javac 11m 22s the patch passed
        +1 checkstyle 0m 32s the patch passed
        +1 mvnsite 1m 0s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 38s the patch passed
        +1 javadoc 0m 49s the patch passed
        +1 unit 9m 5s hadoop-common in the patch passed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        69m 2s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11943
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874438/HDFS-11943.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 956988741bb2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0111711
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20037/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20037/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 17m 41s trunk passed +1 compile 19m 4s trunk passed +1 checkstyle 0m 53s trunk passed +1 mvnsite 1m 35s trunk passed +1 findbugs 1m 38s trunk passed +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 11m 22s the patch passed +1 javac 11m 22s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 1m 0s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 38s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 9m 5s hadoop-common in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 69m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11943 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874438/HDFS-11943.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 956988741bb2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0111711 Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20037/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20037/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Thanks liaoyuxiangqin for the update! The latest patch LGTM and +1.

        Show
        drankye Kai Zheng added a comment - Thanks liaoyuxiangqin for the update! The latest patch LGTM and +1.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 16m 14s trunk passed
        +1 compile 20m 56s trunk passed
        +1 checkstyle 0m 39s trunk passed
        +1 mvnsite 1m 10s trunk passed
        +1 findbugs 1m 33s trunk passed
        +1 javadoc 0m 57s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 15m 45s the patch passed
        +1 javac 15m 45s the patch passed
        +1 checkstyle 0m 40s the patch passed
        +1 mvnsite 1m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 49s the patch passed
        +1 javadoc 1m 2s the patch passed
        -1 unit 9m 22s hadoop-common in the patch failed.
        +1 asflicense 1m 8s The patch does not generate ASF License warnings.
        74m 54s



        Reason Tests
        Failed junit tests hadoop.net.TestClusterTopology



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11943
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874424/HDFS-11943.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2518245f3857 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0111711
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20035/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20035/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20035/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 16m 14s trunk passed +1 compile 20m 56s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 10s trunk passed +1 findbugs 1m 33s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 15m 45s the patch passed +1 javac 15m 45s the patch passed +1 checkstyle 0m 40s the patch passed +1 mvnsite 1m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 1m 2s the patch passed -1 unit 9m 22s hadoop-common in the patch failed. +1 asflicense 1m 8s The patch does not generate ASF License warnings. 74m 54s Reason Tests Failed junit tests hadoop.net.TestClusterTopology Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11943 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874424/HDFS-11943.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2518245f3857 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0111711 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20035/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20035/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20035/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 12m 26s trunk passed
        +1 compile 13m 15s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 1m 4s trunk passed
        -1 findbugs 1m 20s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings.
        +1 javadoc 0m 49s trunk passed
        +1 mvninstall 0m 35s the patch passed
        +1 compile 9m 45s the patch passed
        +1 javac 9m 45s the patch passed
        +1 checkstyle 0m 36s the patch passed
        +1 mvnsite 1m 1s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 27s the patch passed
        +1 javadoc 0m 48s the patch passed
        +1 unit 7m 14s hadoop-common in the patch passed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        52m 51s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11943
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873849/HDFS-11943.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6e05bbc51812 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5db3f98
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19982/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19982/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19982/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 26s trunk passed +1 compile 13m 15s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 4s trunk passed -1 findbugs 1m 20s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. +1 javadoc 0m 49s trunk passed +1 mvninstall 0m 35s the patch passed +1 compile 9m 45s the patch passed +1 javac 9m 45s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 27s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 7m 14s hadoop-common in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 52m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11943 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873849/HDFS-11943.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6e05bbc51812 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5db3f98 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19982/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19982/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19982/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liaoyuxiangqin liaoyuxiangqin added a comment -

        Thanks Kai Zheng review on this. I have override the method in base naitve class
        AbstractNativeRawEncoder/AbstractNativeRawDecoder and which sub class there is no implementation of this method, so all ErasureCoder native implement default use direct buffer and pure java implement default use heap buffer. thanks!

        @Override
          public boolean preferDirectBuffer() {
            return true;
          }
        
        Show
        liaoyuxiangqin liaoyuxiangqin added a comment - Thanks Kai Zheng review on this. I have override the method in base naitve class AbstractNativeRawEncoder/AbstractNativeRawDecoder and which sub class there is no implementation of this method, so all ErasureCoder native implement default use direct buffer and pure java implement default use heap buffer. thanks! @Override public boolean preferDirectBuffer() { return true ; }
        Hide
        drankye Kai Zheng added a comment -

        Thanks liaoyuxiangqin for the update.
        1. Please also check and do the same thing for AbstractNativeRawDecoder. When you ensure the base class to indicate the preference, the method can be removed from sub classes.
        2. Note a minor check style issue.

        Show
        drankye Kai Zheng added a comment - Thanks liaoyuxiangqin for the update. 1. Please also check and do the same thing for AbstractNativeRawDecoder. When you ensure the base class to indicate the preference, the method can be removed from sub classes. 2. Note a minor check style issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 20m 59s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 16m 6s trunk passed
        +1 compile 19m 17s trunk passed
        +1 checkstyle 0m 47s trunk passed
        +1 mvnsite 1m 22s trunk passed
        -1 findbugs 1m 52s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings.
        +1 javadoc 0m 59s trunk passed
        +1 mvninstall 0m 58s the patch passed
        +1 compile 16m 54s the patch passed
        +1 javac 16m 54s the patch passed
        -0 checkstyle 0m 46s hadoop-common-project/hadoop-common: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
        +1 mvnsite 1m 35s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 27s the patch passed
        +1 javadoc 1m 9s the patch passed
        +1 unit 11m 13s hadoop-common in the patch passed.
        +1 asflicense 0m 36s The patch does not generate ASF License warnings.
        98m 20s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11943
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873778/HDFS-11943.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c6921aab5cda 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1a59847
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19975/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19975/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19975/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19975/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 20m 59s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 16m 6s trunk passed +1 compile 19m 17s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 1m 22s trunk passed -1 findbugs 1m 52s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. +1 javadoc 0m 59s trunk passed +1 mvninstall 0m 58s the patch passed +1 compile 16m 54s the patch passed +1 javac 16m 54s the patch passed -0 checkstyle 0m 46s hadoop-common-project/hadoop-common: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) +1 mvnsite 1m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 27s the patch passed +1 javadoc 1m 9s the patch passed +1 unit 11m 13s hadoop-common in the patch passed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 98m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11943 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873778/HDFS-11943.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c6921aab5cda 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1a59847 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19975/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19975/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19975/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19975/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 14m 51s trunk passed
        +1 compile 15m 51s trunk passed
        +1 checkstyle 0m 41s trunk passed
        +1 mvnsite 1m 11s trunk passed
        -1 findbugs 1m 37s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
        +1 javadoc 0m 54s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 12m 39s the patch passed
        +1 javac 12m 39s the patch passed
        -0 checkstyle 0m 40s hadoop-common-project/hadoop-common: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
        +1 mvnsite 1m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 39s the patch passed
        +1 javadoc 0m 49s the patch passed
        -1 unit 8m 0s hadoop-common in the patch failed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        62m 50s



        Reason Tests
        Failed junit tests hadoop.security.TestRaceWhenRelogin



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11943
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873655/HDFS-11943.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a78b6facc911 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 099dfe9
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19967/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19967/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 51s trunk passed +1 compile 15m 51s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 11s trunk passed -1 findbugs 1m 37s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 12m 39s the patch passed +1 javac 12m 39s the patch passed -0 checkstyle 0m 40s hadoop-common-project/hadoop-common: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) +1 mvnsite 1m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 39s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 8m 0s hadoop-common in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 62m 50s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11943 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873655/HDFS-11943.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a78b6facc911 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 099dfe9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19967/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19967/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liaoyuxiangqin liaoyuxiangqin added a comment -

        Thanks for your review and suggestions on this patch Kai Zheng. As you suggestions the NativeXORRawEncoder will default use direct buffer for better performace. And use the PerformanceAdvisory LOG at debug don't affect performance, thanks.

        Show
        liaoyuxiangqin liaoyuxiangqin added a comment - Thanks for your review and suggestions on this patch Kai Zheng . As you suggestions the NativeXORRawEncoder will default use direct buffer for better performace. And use the PerformanceAdvisory LOG at debug don't affect performance, thanks.
        Hide
        drankye Kai Zheng added a comment -

        When you'd update the patch, also note Andrew's above suggestion about PerformanceAdvisory.

        Show
        drankye Kai Zheng added a comment - When you'd update the patch, also note Andrew's above suggestion about PerformanceAdvisory.
        Hide
        drankye Kai Zheng added a comment -

        Thanks liaoyuxiangqin & SammiChen for the discussions!

        Do you have any idea that NativeXORRawEncoder should support direct Buffer or not? I think the answer is yes. Correct me if it's not the case.

        This is correct. Should all native coders indicate they prefer direct bytebuffer for better performance.

        so i think the condition not neccessary to add

        This is correct. Agree.

        So as you guess, the NativeXORRawEncoder doesn't indicate itself support the direct buffer.

        A good reasoning! Actually you got it, that's the cause. Ensuring to have the following block in the base native class AbstractNativeRawEncoder/AbstractNativeRawDecoder should assure all native coders will behave as the thought.

          @Override
          public boolean preferDirectBuffer() {
            return true;
          }
        

        liaoyuxiangqin would you update your patch to fix this? Thanks!

        Show
        drankye Kai Zheng added a comment - Thanks liaoyuxiangqin & SammiChen for the discussions! Do you have any idea that NativeXORRawEncoder should support direct Buffer or not? I think the answer is yes. Correct me if it's not the case. This is correct. Should all native coders indicate they prefer direct bytebuffer for better performance. so i think the condition not neccessary to add This is correct. Agree. So as you guess, the NativeXORRawEncoder doesn't indicate itself support the direct buffer. A good reasoning! Actually you got it, that's the cause. Ensuring to have the following block in the base native class AbstractNativeRawEncoder/AbstractNativeRawDecoder should assure all native coders will behave as the thought. @Override public boolean preferDirectBuffer() { return true ; } liaoyuxiangqin would you update your patch to fix this? Thanks!
        Hide
        liaoyuxiangqin liaoyuxiangqin added a comment -

        Hi SammiChen, thanks for your review. I think the condition

         if (usingDirectBuffer) 
        

        in AbstractNativeRawEncoder.doEncode always false when hadfs use default heap buffer , and the PerformanceAdvisory.LOG.debug will not be able to record info.
        In addition, when use direct buffer the AbstractNativeRawEncoder.doEncode will not be called, and we don't worry about frequent log, so i think the condition not neccessary to add, thanks!

        Show
        liaoyuxiangqin liaoyuxiangqin added a comment - Hi SammiChen , thanks for your review. I think the condition if (usingDirectBuffer) in AbstractNativeRawEncoder.doEncode always false when hadfs use default heap buffer , and the PerformanceAdvisory.LOG.debug will not be able to record info. In addition, when use direct buffer the AbstractNativeRawEncoder.doEncode will not be called, and we don't worry about frequent log, so i think the condition not neccessary to add, thanks!
        Hide
        Sammi SammiChen added a comment -

        Hi liaoyuxiangqin, thanks for provide the detail information. Based on what we know currently, I would suggest that we add the

        if (usingDirectBuffer)

        condition check to correctly popup the warn message at the right time. What do you think?

        Show
        Sammi SammiChen added a comment - Hi liaoyuxiangqin , thanks for provide the detail information. Based on what we know currently, I would suggest that we add the if (usingDirectBuffer) condition check to correctly popup the warn message at the right time. What do you think?
        Hide
        liaoyuxiangqin liaoyuxiangqin added a comment - - edited

        Thanks SammiChen for review on this. The RawErasureEncoder.encode call relation as follows:

        RawErasureEncoder.java
         if (usingDirectBuffer) {
              doEncode(bbeState);
            } else {
              ByteArrayEncodingState baeState = bbeState.convertToByteArrayState();
              doEncode(baeState);  //AbstractNativeRawEncoder.doEncode
            }
        

        After the test, i find hdfs default use heap buffer, so the usingDirectBuffer is false,
        and call AbstractNativeRawEncoder.doEncode print log frequent. In addition, after i modify default value of buffer type, usingDirectBuffer change to true and the call stack is change too and frequent log disappeared.

        So as you guess, the NativeXORRawEncoder doesn't indicate itself support the direct buffer.

        Show
        liaoyuxiangqin liaoyuxiangqin added a comment - - edited Thanks SammiChen for review on this. The RawErasureEncoder.encode call relation as follows: RawErasureEncoder.java if (usingDirectBuffer) { doEncode(bbeState); } else { ByteArrayEncodingState baeState = bbeState.convertToByteArrayState(); doEncode(baeState); //AbstractNativeRawEncoder.doEncode } After the test, i find hdfs default use heap buffer, so the usingDirectBuffer is false, and call AbstractNativeRawEncoder.doEncode print log frequent. In addition, after i modify default value of buffer type, usingDirectBuffer change to true and the call stack is change too and frequent log disappeared. So as you guess, the NativeXORRawEncoder doesn't indicate itself support the direct buffer.
        Hide
        liaoyuxiangqin liaoyuxiangqin added a comment - - edited

        Thanks Andrew Wang for review, i have get the call stack to respond for Kai's question, because hdfs use heap buffer default, so RawErasureEncoder.encode will call AbstractNativeRawEncoder.doEncode, detial encoding stack information as follows:

        WARN rawcoder.AbstractNativeRawEncoder: convertToByteBufferState is invoked, not efficiently. Please use direct ByteBuffer inputs/outputs
        java.lang.Exception: this is ec write log
        at org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawEncoder.doEncode(AbstractNativeRawEncoder.java:69)
        at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureEncoder.encode(RawErasureEncoder.java:87)
        at org.apache.hadoop.hdfs.DFSStripedOutputStream.encode(DFSStripedOutputStream.java:367)
        at org.apache.hadoop.hdfs.DFSStripedOutputStream.writeParityCells(DFSStripedOutputStream.java:909)
        at org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:995)
        at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:829)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:67)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:126)
        at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485)
        at

        In addition, the create Encoder stack as follows:

        at org.apache.hadoop.io.erasurecode.ErasureCodeNative.loadLibrary(Native Method)
        at org.apache.hadoop.io.erasurecode.ErasureCodeNative.<clinit>(ErasureCodeNative.java:46)
        at org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder.<clinit>(NativeXORRawEncoder.java:33)
        at org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.createEncoder(NativeXORRawErasureCoderFactory.java:32)
        at org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:206)
        at org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:152)
        at org.apache.hadoop.hdfs.DFSStripedOutputStream.<init>(DFSStripedOutputStream.java:303)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:305)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1214)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1193)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1131)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:447)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:444)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:458)

        Show
        liaoyuxiangqin liaoyuxiangqin added a comment - - edited Thanks Andrew Wang for review, i have get the call stack to respond for Kai's question, because hdfs use heap buffer default, so RawErasureEncoder.encode will call AbstractNativeRawEncoder.doEncode, detial encoding stack information as follows: WARN rawcoder.AbstractNativeRawEncoder: convertToByteBufferState is invoked, not efficiently. Please use direct ByteBuffer inputs/outputs java.lang.Exception: this is ec write log at org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawEncoder.doEncode(AbstractNativeRawEncoder.java:69) at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureEncoder.encode(RawErasureEncoder.java:87) at org.apache.hadoop.hdfs.DFSStripedOutputStream.encode(DFSStripedOutputStream.java:367) at org.apache.hadoop.hdfs.DFSStripedOutputStream.writeParityCells(DFSStripedOutputStream.java:909) at org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:995) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:829) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:67) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:126) at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485) at In addition, the create Encoder stack as follows: at org.apache.hadoop.io.erasurecode.ErasureCodeNative.loadLibrary(Native Method) at org.apache.hadoop.io.erasurecode.ErasureCodeNative.<clinit>(ErasureCodeNative.java:46) at org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder.<clinit>(NativeXORRawEncoder.java:33) at org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.createEncoder(NativeXORRawErasureCoderFactory.java:32) at org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:206) at org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:152) at org.apache.hadoop.hdfs.DFSStripedOutputStream.<init>(DFSStripedOutputStream.java:303) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:305) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1214) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1193) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1131) at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:447) at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:458)
        Hide
        Sammi SammiChen added a comment -

        Thanks liaoyuxiangqin for working on this. Thanks Andrew Wang for review the patch. I guess the frequent log is because the NativeXORRawEncoder doesn't indicate itself support the direct buffer so far. liaoyuxiangqin, would you please help to verify the guess by using following piece of code?

          if (usingDirectBuffer) {
               PerformanceAdvisory.LOG.debug("convertToByteBufferState is invoked, " + 
               "not efficiently. Please use direct ByteBuffer inputs/outputs");
         }
        

        Kai Zheng Do you have any idea that NativeXORRawEncoder should support direct Buffer or not? I think the answer is yes. Correct me if it's not the case.

        Show
        Sammi SammiChen added a comment - Thanks liaoyuxiangqin for working on this. Thanks Andrew Wang for review the patch. I guess the frequent log is because the NativeXORRawEncoder doesn't indicate itself support the direct buffer so far. liaoyuxiangqin , would you please help to verify the guess by using following piece of code? if (usingDirectBuffer) {  PerformanceAdvisory.LOG.debug("convertToByteBufferState is invoked, " +   "not efficiently. Please use direct ByteBuffer inputs/outputs"); } Kai Zheng Do you have any idea that NativeXORRawEncoder should support direct Buffer or not? I think the answer is yes. Correct me if it's not the case.
        Hide
        andrew.wang Andrew Wang added a comment -

        Patch LGTM, thanks for revving liaoyuxiangqin! Is it possible for you to do a little debugging in your environment to respond to Kai's question?

        Is there any method call stack so we can have an idea why/where on-heap bytebuffers were passed to the native coder? It does affect performance.

        Show
        andrew.wang Andrew Wang added a comment - Patch LGTM, thanks for revving liaoyuxiangqin ! Is it possible for you to do a little debugging in your environment to respond to Kai's question? Is there any method call stack so we can have an idea why/where on-heap bytebuffers were passed to the native coder? It does affect performance.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 12m 59s trunk passed
        +1 compile 13m 38s trunk passed
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 1m 4s trunk passed
        -1 findbugs 1m 24s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
        +1 javadoc 0m 50s trunk passed
        +1 mvninstall 0m 38s the patch passed
        +1 compile 10m 8s the patch passed
        +1 javac 10m 8s the patch passed
        +1 checkstyle 0m 34s the patch passed
        +1 mvnsite 1m 2s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 31s the patch passed
        +1 javadoc 0m 49s the patch passed
        +1 unit 8m 19s hadoop-common in the patch passed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        55m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11943
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872664/HDFS-11943.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux db39da459604 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e86eef9
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19874/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19874/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19874/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 59s trunk passed +1 compile 13m 38s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 4s trunk passed -1 findbugs 1m 24s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 10m 8s the patch passed +1 javac 10m 8s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 31s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 8m 19s hadoop-common in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 55m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11943 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872664/HDFS-11943.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux db39da459604 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e86eef9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19874/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19874/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19874/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liaoyuxiangqin liaoyuxiangqin added a comment - - edited

        Hi, thanks for your review the report Andrew Wang.
        I'd glad to LOG at debug in a more reasonable way, and don't affect performance.

        Show
        liaoyuxiangqin liaoyuxiangqin added a comment - - edited Hi, thanks for your review the report Andrew Wang . I'd glad to LOG at debug in a more reasonable way, and don't affect performance.
        Hide
        drankye Kai Zheng added a comment -

        I think the message itself is very clear about what it means. It complains in doEncode/doDecode regarding the input/output buffers each time. As the call is so frequent I guess we can just remind at the first occurrence time?

        Is there any method call stack so we can have an idea why/where on-heap bytebuffers were passed to the native coder? It does affect performance.

        Show
        drankye Kai Zheng added a comment - I think the message itself is very clear about what it means. It complains in doEncode/doDecode regarding the input/output buffers each time. As the call is so frequent I guess we can just remind at the first occurrence time? Is there any method call stack so we can have an idea why/where on-heap bytebuffers were passed to the native coder? It does affect performance.
        Hide
        andrew.wang Andrew Wang added a comment -

        Ping SammiChen / Kai Zheng also, wondering why this log fires so often in the specified environment?

        Show
        andrew.wang Andrew Wang added a comment - Ping SammiChen / Kai Zheng also, wondering why this log fires so often in the specified environment?
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi, thanks for the JIRA and the patch liaoyuxiangqin!

        One suggestion, do you mind changing this to use the PerformanceAdvisory LOG at debug? e.g. we have these logs for encryption:

          public OpensslSecureRandom() {
            if (!nativeEnabled) {
              PerformanceAdvisory.LOG.debug("Build does not support openssl, " +
                  "falling back to Java SecureRandom.");
              fallback = new java.security.SecureRandom();
            }
          }
        

        We don't need to guard with isDebugEnabled since it's an SLF4J logger.

        Show
        andrew.wang Andrew Wang added a comment - Hi, thanks for the JIRA and the patch liaoyuxiangqin ! One suggestion, do you mind changing this to use the PerformanceAdvisory LOG at debug? e.g. we have these logs for encryption: public OpensslSecureRandom() { if (!nativeEnabled) { PerformanceAdvisory.LOG.debug( "Build does not support openssl, " + "falling back to Java SecureRandom." ); fallback = new java.security.SecureRandom(); } } We don't need to guard with isDebugEnabled since it's an SLF4J logger.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 14m 44s trunk passed
        +1 compile 15m 35s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 1m 5s trunk passed
        -1 findbugs 1m 29s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
        +1 javadoc 0m 51s trunk passed
        +1 mvninstall 0m 43s the patch passed
        +1 compile 10m 34s the patch passed
        +1 javac 10m 34s the patch passed
        +1 checkstyle 0m 34s the patch passed
        +1 mvnsite 1m 4s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 43s the patch passed
        +1 javadoc 0m 51s the patch passed
        +1 unit 7m 45s hadoop-common in the patch passed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        59m 32s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11943
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871806/HDFS-11943.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c675c6fc7456 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5ec7163
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19818/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19818/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19818/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 44s trunk passed +1 compile 15m 35s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 5s trunk passed -1 findbugs 1m 29s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 51s trunk passed +1 mvninstall 0m 43s the patch passed +1 compile 10m 34s the patch passed +1 javac 10m 34s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 43s the patch passed +1 javadoc 0m 51s the patch passed +1 unit 7m 45s hadoop-common in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 59m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11943 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871806/HDFS-11943.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c675c6fc7456 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5ec7163 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19818/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19818/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19818/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            liaoyuxiangqin liaoyuxiangqin
            Reporter:
            liaoyuxiangqin liaoyuxiangqin
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 0.05h
              0.05h
              Remaining:
              Remaining Estimate - 0.05h
              0.05h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development