Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If a disk or controller hardware is faulty then short-circuit read requests can stall indefinitely while reading from the file descriptor. Currently there is no way to detect when short-circuit read requests are slow or blocked.

      This Jira proposes that each BlockReaderLocal maintain read statistics while it is active by measuring the time taken for a pre-determined fraction of read requests. These per-reader stats can be aggregated into global stats when the reader is closed. The aggregate statistics can be exposed via JMX.

      1. HDFS-11789.005.patch
        30 kB
        Hanisha Koneru
      2. HDFS-11789.004.patch
        28 kB
        Hanisha Koneru
      3. HDFS-11789.003.patch
        28 kB
        Hanisha Koneru
      4. HDFS-11789.002.patch
        29 kB
        Hanisha Koneru
      5. HDFS-11789.001.patch
        14 kB
        Hanisha Koneru

        Issue Links

          Activity

          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for contributing this improvement Hanisha Koneru. A few comments:

          1. For maintaining stats one more option is RollingAverages which uses MutableRatesWithAggregation and is optimized for multithreaded updates.
          2. Instead of floating point arithmetic here:
            sampleRangeMax = (int) ((double) conf.getScrMetricsSamplingPercentage()
                    / 100 * Integer.MAX_VALUE);
            

            Alternatively:

            sampleRangeMax = (Integer.MAX_VALUE / 100) * conf.getScrMetricsSamplingPercentage();
            

            Also we should limit getScrMetricsSamplingPercentage to [0, 100] if the administrator misconfigures it.

          3. We should add isolated test cases for BlockReaderIoProvider and BlockReaderLocalMetrics if possible.
          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for contributing this improvement Hanisha Koneru . A few comments: For maintaining stats one more option is RollingAverages which uses MutableRatesWithAggregation and is optimized for multithreaded updates. Instead of floating point arithmetic here: sampleRangeMax = ( int ) (( double ) conf.getScrMetricsSamplingPercentage() / 100 * Integer .MAX_VALUE); Alternatively: sampleRangeMax = ( Integer .MAX_VALUE / 100) * conf.getScrMetricsSamplingPercentage(); Also we should limit getScrMetricsSamplingPercentage to [0, 100] if the administrator misconfigures it. We should add isolated test cases for BlockReaderIoProvider and BlockReaderLocalMetrics if possible.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks for the review Arpit Agarwal. Patch v02 addresses your comments.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks for the review Arpit Agarwal . Patch v02 addresses your comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 36s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 34s Maven dependency ordering for branch
          +1 mvninstall 15m 36s trunk passed
          +1 compile 1m 26s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 29s trunk passed
          +1 findbugs 3m 9s trunk passed
          +1 javadoc 1m 12s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 1m 28s the patch passed
          +1 javac 1m 28s the patch passed
          -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 44 new + 47 unchanged - 0 fixed = 91 total (was 47)
          +1 mvnsite 1m 30s the patch passed
          -1 whitespace 0m 0s The patch 1 line(s) with tabs.
          +1 findbugs 3m 38s the patch passed
          +1 javadoc 1m 2s the patch passed
          +1 unit 1m 14s hadoop-hdfs-client in the patch passed.
          -1 unit 93m 57s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          131m 47s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
            hadoop.hdfs.server.datanode.TestDataNodeUUID



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11789
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872890/HDFS-11789.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 89a80f177256 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8633ef8
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19900/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19900/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19900/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19900/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19900/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 36s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 34s Maven dependency ordering for branch +1 mvninstall 15m 36s trunk passed +1 compile 1m 26s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 29s trunk passed +1 findbugs 3m 9s trunk passed +1 javadoc 1m 12s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 1m 28s the patch passed +1 javac 1m 28s the patch passed -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 44 new + 47 unchanged - 0 fixed = 91 total (was 47) +1 mvnsite 1m 30s the patch passed -1 whitespace 0m 0s The patch 1 line(s) with tabs. +1 findbugs 3m 38s the patch passed +1 javadoc 1m 2s the patch passed +1 unit 1m 14s hadoop-hdfs-client in the patch passed. -1 unit 93m 57s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 131m 47s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.server.datanode.TestDataNodeUUID Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11789 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872890/HDFS-11789.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 89a80f177256 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8633ef8 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19900/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19900/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19900/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19900/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19900/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Thanks for updating the patch Hanisha Koneru. A few comments:

          1. Typos in SHORT_CIRCUIT_READ_LATECNY_METRIC_REGISTERD_NAME.
          2. The test-only method getShortCircuitReadRollingAverages should be tagged with @VisibleForTesting. Also it should be package-private if possible.
          3. I think we can eliminate the METRICS_ENABLED config key. We can enable the metric when the sampling percentage is > 0.
          4. Let's avoid static initialization of BlockReaderLocal#metrics. One issue is that adding the overhead of a MutableRollingAverages object (and its rates roller thread) for all clients, most of which may will never enable SCR statistics. I think your original approach of initializing it within a lock on construction was fine, the overhead of that lock compared to the overhead of cloning a file descriptor via system calls should be minimal.

          Still reviewing the tests.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Thanks for updating the patch Hanisha Koneru . A few comments: Typos in SHORT_CIRCUIT_READ_LATECNY_METRIC_REGISTERD_NAME . The test-only method getShortCircuitReadRollingAverages should be tagged with @VisibleForTesting . Also it should be package-private if possible. I think we can eliminate the METRICS_ENABLED config key. We can enable the metric when the sampling percentage is > 0. Let's avoid static initialization of BlockReaderLocal#metrics . One issue is that adding the overhead of a MutableRollingAverages object (and its rates roller thread) for all clients, most of which may will never enable SCR statistics. I think your original approach of initializing it within a lock on construction was fine, the overhead of that lock compared to the overhead of cloning a file descriptor via system calls should be minimal. Still reviewing the tests.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks Arpit Agarwal for the review. Posted patch v03 with the updates.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks Arpit Agarwal for the review. Posted patch v03 with the updates.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 24s Maven dependency ordering for branch
          +1 mvninstall 13m 9s trunk passed
          +1 compile 1m 24s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 findbugs 2m 58s trunk passed
          +1 javadoc 1m 2s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 1m 22s the patch passed
          +1 javac 1m 22s the patch passed
          -0 checkstyle 0m 39s hadoop-hdfs-project: The patch generated 28 new + 47 unchanged - 0 fixed = 75 total (was 47)
          +1 mvnsite 1m 21s the patch passed
          -1 whitespace 0m 0s The patch 1 line(s) with tabs.
          -1 findbugs 1m 29s hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 58s the patch passed
          +1 unit 1m 11s hadoop-hdfs-client in the patch passed.
          -1 unit 64m 50s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          98m 3s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client
            new org.apache.hadoop.hdfs.client.impl.BlockReaderLocal(BlockReaderLocal$Builder) does not release lock on all exception paths At BlockReaderLocal.java:lock on all exception paths At BlockReaderLocal.java:[line 288]
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11789
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873050/HDFS-11789.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 87aa177b5bd7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 999c8fc
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19910/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19910/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19910/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19910/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19910/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19910/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 24s Maven dependency ordering for branch +1 mvninstall 13m 9s trunk passed +1 compile 1m 24s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 26s trunk passed +1 findbugs 2m 58s trunk passed +1 javadoc 1m 2s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 22s the patch passed +1 javac 1m 22s the patch passed -0 checkstyle 0m 39s hadoop-hdfs-project: The patch generated 28 new + 47 unchanged - 0 fixed = 75 total (was 47) +1 mvnsite 1m 21s the patch passed -1 whitespace 0m 0s The patch 1 line(s) with tabs. -1 findbugs 1m 29s hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 58s the patch passed +1 unit 1m 11s hadoop-hdfs-client in the patch passed. -1 unit 64m 50s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 98m 3s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client   new org.apache.hadoop.hdfs.client.impl.BlockReaderLocal(BlockReaderLocal$Builder) does not release lock on all exception paths At BlockReaderLocal.java:lock on all exception paths At BlockReaderLocal.java: [line 288] Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11789 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873050/HDFS-11789.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 87aa177b5bd7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 999c8fc Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19910/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19910/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19910/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/19910/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19910/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19910/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Thanks for updating the patch Hanisha Koneru. A few more comments:

          1. In BlockReaderLocal constructor, metrics is being passed to BlockReaderIoProvider before it is potentially initialized.
                this.blockReaderIoProvider = new BlockReaderIoProvider(
                    builder.shortCircuitConf, metrics, timer);
            
                if (builder.shortCircuitConf.isScrMetricsEnabled()) {
                  metricsInitializationLock.lock();
                  if (!isMetricsEnabled) {
                    metrics = BlockReaderLocalMetrics.create();
                    isMetricsEnabled = true;
                  }
            
          2. Typo - SHORT_CIRCUIT_READ_LATECNY_METRIC_REGISTERD_NAME.
          3. Looks like we can eliminate the BlockReaderLocal#isMetricsEnabled, instead just check whether metrics is null.
          4. Thanks for adding TestBlockReaderIoProvider. One more related test - ensure addShortCircuitReadLatency is not invoked when the read takes less than the threshold.
          5. Also I think we can make BlockReaderIoProvider be a static utility class since it doesn't maintain any mutable state. But this approach is also fine since the allocated object is lightweight.

          TestBlockReaderLocalMetrics is a nicely written unit test!

          Edit: Sorry about the confusion, there is no threshold for addShortCircuitReadLatency.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Thanks for updating the patch Hanisha Koneru . A few more comments: In BlockReaderLocal constructor, metrics is being passed to BlockReaderIoProvider before it is potentially initialized. this .blockReaderIoProvider = new BlockReaderIoProvider( builder.shortCircuitConf, metrics, timer); if (builder.shortCircuitConf.isScrMetricsEnabled()) { metricsInitializationLock.lock(); if (!isMetricsEnabled) { metrics = BlockReaderLocalMetrics.create(); isMetricsEnabled = true ; } Typo - SHORT_CIRCUIT_READ_LATECNY_METRIC_REGISTERD_NAME . Looks like we can eliminate the BlockReaderLocal#isMetricsEnabled , instead just check whether metrics is null. Thanks for adding TestBlockReaderIoProvider. One more related test - ensure addShortCircuitReadLatency is not invoked when the read takes less than the threshold. Also I think we can make BlockReaderIoProvider be a static utility class since it doesn't maintain any mutable state. But this approach is also fine since the allocated object is lightweight. TestBlockReaderLocalMetrics is a nicely written unit test! Edit: Sorry about the confusion, there is no threshold for addShortCircuitReadLatency.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thank you Arpit Agarwal.
          Patch v04 addresses your comments, checkstyle and findbug errors.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal . Patch v04 addresses your comments, checkstyle and findbug errors.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Hanisha Koneru. +1 pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Hanisha Koneru . +1 pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 14m 18s trunk passed
          +1 compile 1m 40s trunk passed
          +1 checkstyle 0m 46s trunk passed
          +1 mvnsite 1m 38s trunk passed
          +1 findbugs 3m 27s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 1m 54s the patch passed
          +1 javac 1m 54s the patch passed
          -0 checkstyle 0m 47s hadoop-hdfs-project: The patch generated 1 new + 47 unchanged - 0 fixed = 48 total (was 47)
          +1 mvnsite 1m 34s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 40s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 1m 13s hadoop-hdfs-client in the patch passed.
          -1 unit 66m 40s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          103m 37s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11789
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873924/HDFS-11789.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 013b3da8619d 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e806c6e
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19985/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19985/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19985/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19985/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 14m 18s trunk passed +1 compile 1m 40s trunk passed +1 checkstyle 0m 46s trunk passed +1 mvnsite 1m 38s trunk passed +1 findbugs 3m 27s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 1m 54s the patch passed +1 javac 1m 54s the patch passed -0 checkstyle 0m 47s hadoop-hdfs-project: The patch generated 1 new + 47 unchanged - 0 fixed = 48 total (was 47) +1 mvnsite 1m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 40s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 1m 13s hadoop-hdfs-client in the patch passed. -1 unit 66m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 103m 37s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11789 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873924/HDFS-11789.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 013b3da8619d 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e806c6e Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19985/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19985/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19985/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19985/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the patch revisions Hanisha Koneru!

          Can you please add a package-info.java file to address the remaining checkstyle issue? You can take a look at the following package-info files as examples:

          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/startupprogress/package-info.java
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/package-info.java
          
          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the patch revisions Hanisha Koneru ! Can you please add a package-info.java file to address the remaining checkstyle issue? You can take a look at the following package-info files as examples: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/startupprogress/ package -info.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/ package -info.java
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks for the review Arpit Agarwal. I added the package-info in patch v05.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks for the review Arpit Agarwal . I added the package-info in patch v05.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the v05 patch, pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the v05 patch, pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 13m 8s trunk passed
          +1 compile 1m 25s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 findbugs 3m 1s trunk passed
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 1m 23s the patch passed
          +1 javac 1m 23s the patch passed
          +1 checkstyle 0m 38s the patch passed
          +1 mvnsite 1m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 12s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 1m 10s hadoop-hdfs-client in the patch passed.
          -1 unit 69m 30s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          102m 23s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.qjournal.client.TestQuorumJournalManager
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11789
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874120/HDFS-11789.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 78f56293030c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8dbd53e
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20000/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20000/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20000/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 13m 8s trunk passed +1 compile 1m 25s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 27s trunk passed +1 findbugs 3m 1s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 23s the patch passed +1 javac 1m 23s the patch passed +1 checkstyle 0m 38s the patch passed +1 mvnsite 1m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 12s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 1m 10s hadoop-hdfs-client in the patch passed. -1 unit 69m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 102m 23s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11789 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874120/HDFS-11789.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 78f56293030c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8dbd53e Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20000/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20000/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20000/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          All the failed junit tests pass locally.

          Show
          hanishakoneru Hanisha Koneru added a comment - All the failed junit tests pass locally.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I've committed this. Thank you for the contribution Hanisha Koneru!

          Show
          arpitagarwal Arpit Agarwal added a comment - I've committed this. Thank you for the contribution Hanisha Koneru !
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11909 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11909/)
          HDFS-11789. Maintain Short-Circuit Read Statistics. Contributed by (arp: rev 6d116ffad23b470f8e9ca131d8e89cbbbb4378d7)

          • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestBlockReaderIoProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderLocal.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestBlockReaderLocalMetrics.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/metrics/BlockReaderLocalMetrics.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/metrics/package-info.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/metrics/BlockReaderIoProvider.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11909 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11909/ ) HDFS-11789 . Maintain Short-Circuit Read Statistics. Contributed by (arp: rev 6d116ffad23b470f8e9ca131d8e89cbbbb4378d7) (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestBlockReaderIoProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderLocal.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestBlockReaderLocalMetrics.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/metrics/BlockReaderLocalMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/metrics/package-info.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/metrics/BlockReaderIoProvider.java

            People

            • Assignee:
              hanishakoneru Hanisha Koneru
              Reporter:
              hanishakoneru Hanisha Koneru
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development