Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: hdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-10958 introduces instrumentation hooks around DataNode disk IO and HDFS-10959 adds support for profiling hooks to expose latency statistics.
      Instead of choosing only one hook using Config parameters, we want to add two separate hooks - one for profiling and one for fault injection. The fault injection hook will be useful for testing purposes.
      This jira only introduces support for fault injection hook. The implementation for that will come later on.
      Also, now Default and Counting FileIOEvents would not be needed as we can control enabling the profiling and fault injection hooks using config parameters.

      1. HDFS-11299.000.patch
        57 kB
        Hanisha Koneru
      2. HDFS-11299.001.patch
        56 kB
        Hanisha Koneru
      3. HDFS-11299.002.patch
        57 kB
        Hanisha Koneru

        Activity

        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Cherry-picked to branch-2.

        Show
        arpitagarwal Arpit Agarwal added a comment - Cherry-picked to branch-2.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11101 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11101/)
        HDFS-11299. Support multiple Datanode File IO hooks. Contributed by (arp: rev 4046794a5365f80f9fa002e3889e41c6d29e13a8)

        • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/CountingFileIoEvents.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FileIoEvents.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FileIoProvider.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeMXBean.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FaultInjectorFileIoEvents.java
        • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DefaultFileIoEvents.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeMetrics.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11101 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11101/ ) HDFS-11299 . Support multiple Datanode File IO hooks. Contributed by (arp: rev 4046794a5365f80f9fa002e3889e41c6d29e13a8) (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/CountingFileIoEvents.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FileIoEvents.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FileIoProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeMXBean.java (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FaultInjectorFileIoEvents.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DefaultFileIoEvents.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java
        Hide
        arpitagarwal Arpit Agarwal added a comment - - edited

        The test case failure is unrelated and the checkstyle warning can be ignored (the 'unused' import is referenced from a Javadoc comment).

        Show
        arpitagarwal Arpit Agarwal added a comment - - edited The test case failure is unrelated and the checkstyle warning can be ignored (the 'unused' import is referenced from a Javadoc comment).
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Committed to trunk. Thanks for the contribution Hanisha Koneru and thanks for the review Xiaoyu Yao.

        Show
        arpitagarwal Arpit Agarwal added a comment - Committed to trunk. Thanks for the contribution Hanisha Koneru and thanks for the review Xiaoyu Yao .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 2m 10s Maven dependency ordering for branch
        +1 mvninstall 13m 38s trunk passed
        +1 compile 10m 11s trunk passed
        +1 checkstyle 1m 42s trunk passed
        +1 mvnsite 2m 4s trunk passed
        +1 mvneclipse 0m 38s trunk passed
        +1 findbugs 3m 27s trunk passed
        +1 javadoc 1m 33s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 10m 3s the patch passed
        +1 javac 10m 3s the patch passed
        -0 checkstyle 1m 45s root: The patch generated 1 new + 578 unchanged - 0 fixed = 579 total (was 578)
        +1 mvnsite 2m 11s the patch passed
        +1 mvneclipse 0m 38s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 48s the patch passed
        +1 javadoc 1m 37s the patch passed
        +1 unit 7m 55s hadoop-common in the patch passed.
        -1 unit 68m 5s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 37s The patch does not generate ASF License warnings.
        135m 14s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11299
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846413/HDFS-11299.002.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux 8cb3224826df 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 91bf504
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18115/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18115/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18115/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18115/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 10s Maven dependency ordering for branch +1 mvninstall 13m 38s trunk passed +1 compile 10m 11s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 2m 4s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 3m 27s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 10m 3s the patch passed +1 javac 10m 3s the patch passed -0 checkstyle 1m 45s root: The patch generated 1 new + 578 unchanged - 0 fixed = 579 total (was 578) +1 mvnsite 2m 11s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 48s the patch passed +1 javadoc 1m 37s the patch passed +1 unit 7m 55s hadoop-common in the patch passed. -1 unit 68m 5s hadoop-hdfs in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 135m 14s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11299 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846413/HDFS-11299.002.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 8cb3224826df 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 91bf504 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18115/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18115/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18115/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18115/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Hanisha Koneru for updating the patch and Arpit Agarwal for the review.

        Looks like LEN_INT can be replaced with Integer.BYTES from JDK. But Integer.BYTES is only available in JDK1.8. It will cause more work when backporting it to branch-2.
        I'm OK with the patch v02. +1 pending Jenkins.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Hanisha Koneru for updating the patch and Arpit Agarwal for the review. Looks like LEN_INT can be replaced with Integer.BYTES from JDK. But Integer.BYTES is only available in JDK1.8. It will cause more work when backporting it to branch-2. I'm OK with the patch v02. +1 pending Jenkins.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks Hanisha Koneru.

        +1 pending Jenkins. Looks like Xiaoyu's feedback is also addressed but will hold off committing until tomorrow in case he has more feedback.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks Hanisha Koneru . +1 pending Jenkins. Looks like Xiaoyu's feedback is also addressed but will hold off committing until tomorrow in case he has more feedback.
        Hide
        hanishakoneru Hanisha Koneru added a comment -

        Thank you Xiaoyu Yao and Arpit Agarwal for reviewing the patch and for the comments. I have addressed them in patch v02.

        Show
        hanishakoneru Hanisha Koneru added a comment - Thank you Xiaoyu Yao and Arpit Agarwal for reviewing the patch and for the comments. I have addressed them in patch v02.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thank you for the updated patch Hanisha Koneru. In addition to Xiaoyu's feedback, one typo in the setting name: dfs.datanode.enable.fileio.fault.injectio. injectio --> injection.

        The length field is a pre-existing bug but it's a good to fix it.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thank you for the updated patch Hanisha Koneru . In addition to Xiaoyu's feedback, one typo in the setting name: dfs.datanode.enable.fileio.fault.injectio. injectio --> injection . The length field is a pre-existing bug but it's a good to fix it.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Hanisha Koneru for working on this. The latest patch looks pretty good to me. I just have few comments.

        1. Metrics.md Line 328
        `dfs.datanode.fileio.events.enabled.profiling` should be `dfs.datanode.enable.fileio.profiling` to match with the code.

        2. FaultInjectorFileIoEvents.java: checkstyle issues, looks like the indentation is not setup properly in your editor.

        3. FileIoProvider.java Line 819/821/823/898/900/902/959/961/963/1002/1004/1006
        NIT: should we count read/write of a int as 4 bytes, i.e. using sizeof(int) instead of 1 here to be consistent thought the len field is not being used by the current provider implementation?

        Show
        xyao Xiaoyu Yao added a comment - Thanks Hanisha Koneru for working on this. The latest patch looks pretty good to me. I just have few comments. 1. Metrics.md Line 328 `dfs.datanode.fileio.events.enabled.profiling` should be `dfs.datanode.enable.fileio.profiling` to match with the code. 2. FaultInjectorFileIoEvents.java: checkstyle issues, looks like the indentation is not setup properly in your editor. 3. FileIoProvider.java Line 819/821/823/898/900/902/959/961/963/1002/1004/1006 NIT: should we count read/write of a int as 4 bytes, i.e. using sizeof(int) instead of 1 here to be consistent thought the len field is not being used by the current provider implementation?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 1m 45s Maven dependency ordering for branch
        +1 mvninstall 12m 49s trunk passed
        +1 compile 10m 4s trunk passed
        +1 checkstyle 1m 40s trunk passed
        +1 mvnsite 2m 3s trunk passed
        +1 mvneclipse 0m 37s trunk passed
        +1 findbugs 3m 23s trunk passed
        +1 javadoc 1m 34s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 34s the patch passed
        +1 compile 10m 15s the patch passed
        +1 javac 10m 15s the patch passed
        -0 checkstyle 1m 45s root: The patch generated 33 new + 578 unchanged - 1 fixed = 611 total (was 579)
        +1 mvnsite 2m 8s the patch passed
        +1 mvneclipse 0m 38s the patch passed
        -1 whitespace 0m 0s The patch 16 line(s) with tabs.
        +1 findbugs 4m 0s the patch passed
        +1 javadoc 1m 58s the patch passed
        -1 unit 7m 45s hadoop-common in the patch failed.
        +1 unit 67m 59s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 38s The patch does not generate ASF License warnings.
        134m 17s



        Reason Tests
        Failed junit tests hadoop.net.TestClusterTopology



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11299
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846143/HDFS-11299.001.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux 9d62bb0af2ce 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 71a4acf
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18100/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18100/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18100/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18100/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18100/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 45s Maven dependency ordering for branch +1 mvninstall 12m 49s trunk passed +1 compile 10m 4s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 2m 3s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 3m 23s trunk passed +1 javadoc 1m 34s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 10m 15s the patch passed +1 javac 10m 15s the patch passed -0 checkstyle 1m 45s root: The patch generated 33 new + 578 unchanged - 1 fixed = 611 total (was 579) +1 mvnsite 2m 8s the patch passed +1 mvneclipse 0m 38s the patch passed -1 whitespace 0m 0s The patch 16 line(s) with tabs. +1 findbugs 4m 0s the patch passed +1 javadoc 1m 58s the patch passed -1 unit 7m 45s hadoop-common in the patch failed. +1 unit 67m 59s hadoop-hdfs in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 134m 17s Reason Tests Failed junit tests hadoop.net.TestClusterTopology Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11299 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846143/HDFS-11299.001.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 9d62bb0af2ce 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 71a4acf Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18100/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18100/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18100/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18100/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18100/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 1m 59s Maven dependency ordering for branch
        +1 mvninstall 13m 7s trunk passed
        +1 compile 10m 21s trunk passed
        +1 checkstyle 1m 41s trunk passed
        +1 mvnsite 2m 5s trunk passed
        +1 mvneclipse 0m 38s trunk passed
        +1 findbugs 3m 23s trunk passed
        +1 javadoc 1m 36s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 27s the patch passed
        +1 compile 9m 38s the patch passed
        +1 javac 9m 38s the patch passed
        -0 checkstyle 1m 42s root: The patch generated 33 new + 579 unchanged - 0 fixed = 612 total (was 579)
        +1 mvnsite 2m 0s the patch passed
        +1 mvneclipse 0m 37s the patch passed
        -1 whitespace 0m 0s The patch 16 line(s) with tabs.
        +1 findbugs 3m 40s the patch passed
        +1 javadoc 1m 35s the patch passed
        +1 unit 7m 49s hadoop-common in the patch passed.
        +1 unit 87m 2s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        152m 36s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11299
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846143/HDFS-11299.001.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux 57cc8a608c24 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 71a4acf
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18098/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18098/artifact/patchprocess/whitespace-tabs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18098/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18098/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 59s Maven dependency ordering for branch +1 mvninstall 13m 7s trunk passed +1 compile 10m 21s trunk passed +1 checkstyle 1m 41s trunk passed +1 mvnsite 2m 5s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 3m 23s trunk passed +1 javadoc 1m 36s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 9m 38s the patch passed +1 javac 9m 38s the patch passed -0 checkstyle 1m 42s root: The patch generated 33 new + 579 unchanged - 0 fixed = 612 total (was 579) +1 mvnsite 2m 0s the patch passed +1 mvneclipse 0m 37s the patch passed -1 whitespace 0m 0s The patch 16 line(s) with tabs. +1 findbugs 3m 40s the patch passed +1 javadoc 1m 35s the patch passed +1 unit 7m 49s hadoop-common in the patch passed. +1 unit 87m 2s hadoop-hdfs in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 152m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11299 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846143/HDFS-11299.001.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 57cc8a608c24 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 71a4acf Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18098/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18098/artifact/patchprocess/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18098/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18098/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hanishakoneru Hanisha Koneru added a comment -

        Thank you Arpit Agarwal for the review. I have addressed your comments in patch v01.

        Show
        hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal for the review. I have addressed your comments in patch v01.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks for this improvement Hanisha Koneru. Comments below:

        1. FaultInjectorFileIoEvents#beforeMetadataOp and FaultInjectorFileIoEvents#beforeFileIo can return void.
        2. We can just remove FileIoProvider#getStatistics, also FaultInjectorFileIoEvents#getStatistics and Datanode#getFileIoProviderStatistics.
        3. We should keep the DiskChecker invocation out of ProfilingFileIoEvents. One way is to add a failure handler in FileIoProvider which invokes diskchecker then invokes ProfilingFileIoEvents#onFailure.
        4. Minor: The config key names can be simplified. e.g. dfs.datanode.fileio.events.enabled.profiling ==> dfs.datanode.enable.fileio.profiling.

        Looks good otherwise.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks for this improvement Hanisha Koneru . Comments below: FaultInjectorFileIoEvents#beforeMetadataOp and FaultInjectorFileIoEvents#beforeFileIo can return void. We can just remove FileIoProvider#getStatistics, also FaultInjectorFileIoEvents#getStatistics and Datanode#getFileIoProviderStatistics. We should keep the DiskChecker invocation out of ProfilingFileIoEvents. One way is to add a failure handler in FileIoProvider which invokes diskchecker then invokes ProfilingFileIoEvents#onFailure. Minor: The config key names can be simplified. e.g. dfs.datanode.fileio.events.enabled.profiling ==> dfs.datanode.enable.fileio.profiling . Looks good otherwise.

          People

          • Assignee:
            hanishakoneru Hanisha Koneru
            Reporter:
            hanishakoneru Hanisha Koneru
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development