Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10876

Dispatcher#dispatch should log IOException stacktrace

    Details

      Description

      This error logging should be improved. A warning log should record the stacktrace as well.

      Dispatcher#dispatch
      try {
      ...
      } catch (IOException e) {
              LOG.warn("Failed to move " + this + ": " + e.getMessage());
      ...
      }
      
      1. HDFS-10876.001.patch
        1.0 kB
        Manoj Govindassamy

        Activity

        Hide
        manojg Manoj Govindassamy added a comment -

        Attached v01 patch to log the exception stacktrace in Dispatcher#dispatch.

        Show
        manojg Manoj Govindassamy added a comment - Attached v01 patch to log the exception stacktrace in Dispatcher#dispatch.
        Hide
        liuml07 Mingliang Liu added a comment -

        +1 pending on Jenkins.

        Show
        liuml07 Mingliang Liu added a comment - +1 pending on Jenkins.
        Hide
        liuml07 Mingliang Liu added a comment -

        I'm also thinking of logging the exception stack trace in INFO or DEBUG level.

        Show
        liuml07 Mingliang Liu added a comment - I'm also thinking of logging the exception stack trace in INFO or DEBUG level.
        Hide
        manojg Manoj Govindassamy added a comment -

        Thanks for the review. Exception is because of the proxy or target issue during balancing. Upon these issues, 10 second delay for proxy and target are activated. So, would logging this stacktrace as an INFO message be OK ? But, I also see your point of not spamming the logs with stack traces and may be DEBUG for stack trace sounds like a good idea.

        Wei-Chiu Chuang any thoughts please ?

        Show
        manojg Manoj Govindassamy added a comment - Thanks for the review. Exception is because of the proxy or target issue during balancing. Upon these issues, 10 second delay for proxy and target are activated. So, would logging this stacktrace as an INFO message be OK ? But, I also see your point of not spamming the logs with stack traces and may be DEBUG for stack trace sounds like a good idea. Wei-Chiu Chuang any thoughts please ?
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        INFO may be fine.
        However, I am seeing an IOE in balancer that disrupts normal balancer operation. It only occurs after several hours of balancing.

        2016-05-14 08:56:27,636 WARN [pool-2302-thread-7] balancer.Dispatcher (Dispatcher.java:dispatch(325)) - Failed to move blk_1550828018_1100035582559 with size=167 from 10.17.1.1:1004:DISK to 10.17.1.2:1004:DISK through 10.17.1.1:1004: Can't re-compute encryption key for nonce, since the required block key (keyID=-327392027) doesn't exist. Current key: -327392024

        Balancer aborts after five iterations if no blocks are moved. If you make it DEBUG it's going to be really hard to tell what's going on. From a support perspective it's going to generated many support tickets.

        Show
        jojochuang Wei-Chiu Chuang added a comment - INFO may be fine. However, I am seeing an IOE in balancer that disrupts normal balancer operation. It only occurs after several hours of balancing. 2016-05-14 08:56:27,636 WARN [pool-2302-thread-7] balancer.Dispatcher (Dispatcher.java:dispatch(325)) - Failed to move blk_1550828018_1100035582559 with size=167 from 10.17.1.1:1004:DISK to 10.17.1.2:1004:DISK through 10.17.1.1:1004: Can't re-compute encryption key for nonce, since the required block key (keyID=-327392027) doesn't exist. Current key: -327392024 Balancer aborts after five iterations if no blocks are moved. If you make it DEBUG it's going to be really hard to tell what's going on. From a support perspective it's going to generated many support tickets.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 49s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 49s trunk passed
        +1 javadoc 1m 1s trunk passed
        +1 mvninstall 0m 56s the patch passed
        +1 compile 0m 52s the patch passed
        +1 javac 0m 52s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 0s the patch passed
        +1 javadoc 1m 1s the patch passed
        +1 unit 66m 15s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        86m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10876
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829424/HDFS-10876.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4285ffe72eff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e45307c
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16810/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16810/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 49s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 0m 56s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 66m 15s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 86m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10876 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829424/HDFS-10876.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4285ffe72eff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e45307c Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16810/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16810/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hello Mingliang Liu, any other comments on log level? Thanks!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hello Mingliang Liu , any other comments on log level? Thanks!
        Hide
        liuml07 Mingliang Liu added a comment -

        It makes sense. Thanks.

        Show
        liuml07 Mingliang Liu added a comment - It makes sense. Thanks.
        Hide
        manojg Manoj Govindassamy added a comment -

        Thanks for the review Wei-Chiu Chuang and Mingliang Liu.

        Show
        manojg Manoj Govindassamy added a comment - Thanks for the review Wei-Chiu Chuang and Mingliang Liu .
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed the 001 patch to trunk, branch-2 and branch-2.8. Thanks Manoj Govindassamy for contributing the patch and thanks Mingliang Liu for the comments!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed the 001 patch to trunk, branch-2 and branch-2.8. Thanks Manoj Govindassamy for contributing the patch and thanks Mingliang Liu for the comments!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10481 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10481/)
        HDFS-10876. Dispatcher#dispatch should log IOException stacktrace. (weichiu: rev 74b3dd514c86b46197e2e19d9824a423715cab30)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10481 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10481/ ) HDFS-10876 . Dispatcher#dispatch should log IOException stacktrace. (weichiu: rev 74b3dd514c86b46197e2e19d9824a423715cab30) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java

          People

          • Assignee:
            manojg Manoj Govindassamy
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development