Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12795

KMS does not log detailed stack trace for unexpected errors.

    Details

    • Target Version/s:

      Description

      If the KMS server encounters an unexpected error resulting in an HTTP 500 response, it does not log the stack trace. This makes it difficult to troubleshoot. The client side exception cannot provide further details.

      1. HADOOP-12795.001.patch
        0.8 kB
        Chris Nauroth
      2. HADOOP-12795.002.patch
        1 kB
        Chris Nauroth

        Activity

        Hide
        cnauroth Chris Nauroth added a comment -

        It appears there was some consideration of logging stack traces at some point. There is a KMSExceptionsProvider#log method that would log the stack trace, but I can't find any code that calls that method. Maybe we just need to add a call to that method when there is an internal server error. (See attached patch.)

        Arun Suresh, could you please comment? Thank you.

        Show
        cnauroth Chris Nauroth added a comment - It appears there was some consideration of logging stack traces at some point. There is a KMSExceptionsProvider#log method that would log the stack trace, but I can't find any code that calls that method. Maybe we just need to add a call to that method when there is an internal server error. (See attached patch.) Arun Suresh , could you please comment? Thank you.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hi Chris Nauroth, I've recently encountered similar issue with KMS returning code 500, and it would be great to add this log. We ended up looking at kms-catalina.log and found indirect evidence to the root cause. Have you checked that log too?

        As for the patch, the log you added seem to be used in test only, not in production.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hi Chris Nauroth , I've recently encountered similar issue with KMS returning code 500, and it would be great to add this log. We ended up looking at kms-catalina.log and found indirect evidence to the root cause. Have you checked that log too? As for the patch, the log you added seem to be used in test only, not in production.
        Hide
        cnauroth Chris Nauroth added a comment -

        Wei-Chiu Chuang, thank you for your reply.

        We ended up looking at kms-catalina.log and found indirect evidence to the root cause. Have you checked that log too?

        Thanks for the suggestion, but in my case, the catalina logs were not sufficient either. There is a record of the error in kms-audit.log, and you can also enable request/response logging in kms.log by editing web.xml. Both of these show the presence of an error and the short message, but nothing provides the full stack trace for a deeper look.

        As for the patch, the log you added seem to be used in test only, not in production.

        I applied my patch and simulated an error by hard-coding a method to throw an unchecked exception. With that, I could see the full stack trace in kms.log, so I think the patch is working for the production code.

        Show
        cnauroth Chris Nauroth added a comment - Wei-Chiu Chuang , thank you for your reply. We ended up looking at kms-catalina.log and found indirect evidence to the root cause. Have you checked that log too? Thanks for the suggestion, but in my case, the catalina logs were not sufficient either. There is a record of the error in kms-audit.log, and you can also enable request/response logging in kms.log by editing web.xml. Both of these show the presence of an error and the short message, but nothing provides the full stack trace for a deeper look. As for the patch, the log you added seem to be used in test only, not in production. I applied my patch and simulated an error by hard-coding a method to throw an unchecked exception. With that, I could see the full stack trace in kms.log, so I think the patch is working for the production code.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 33s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 19s trunk passed
        +1 compile 8m 3s trunk passed with JDK v1.8.0_72
        +1 compile 7m 14s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 20s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 26s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 13s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 19s the patch passed
        +1 compile 7m 57s the patch passed with JDK v1.8.0_72
        +1 javac 7m 57s the patch passed
        +1 compile 7m 11s the patch passed with JDK v1.7.0_95
        +1 javac 7m 11s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 0m 19s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 36s the patch passed
        +1 javadoc 0m 13s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
        +1 unit 1m 33s hadoop-kms in the patch passed with JDK v1.8.0_72.
        +1 unit 1m 35s hadoop-kms in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        46m 25s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787524/HADOOP-12795.001.patch
        JIRA Issue HADOOP-12795
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5e83586de1e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 23f937e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8603/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Max memory used 77MB
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8603/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 19s trunk passed +1 compile 8m 3s trunk passed with JDK v1.8.0_72 +1 compile 7m 14s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 13s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 19s the patch passed +1 compile 7m 57s the patch passed with JDK v1.8.0_72 +1 javac 7m 57s the patch passed +1 compile 7m 11s the patch passed with JDK v1.7.0_95 +1 javac 7m 11s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 36s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 1m 33s hadoop-kms in the patch passed with JDK v1.8.0_72. +1 unit 1m 35s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 46m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787524/HADOOP-12795.001.patch JIRA Issue HADOOP-12795 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5e83586de1e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23f937e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8603/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8603/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Chris Nauroth, Thanks for raising this..

        Yup, the KMSExceptionsProvider#log method was written specifically for that purpose.. must've missed wiring it in..
        Your patch should work.. maybe you can also add the log to the other INTERNAL_SERVER_ERROR clause as well ?
        +1 pending that

        Show
        asuresh Arun Suresh added a comment - Chris Nauroth , Thanks for raising this.. Yup, the KMSExceptionsProvider#log method was written specifically for that purpose.. must've missed wiring it in.. Your patch should work.. maybe you can also add the log to the other INTERNAL_SERVER_ERROR clause as well ? +1 pending that
        Hide
        cnauroth Chris Nauroth added a comment -

        The patch doesn't appear to include any new or modified tests.

        This is a change in logging only, which I have verified manually as per my last comment.

        Show
        cnauroth Chris Nauroth added a comment - The patch doesn't appear to include any new or modified tests. This is a change in logging only, which I have verified manually as per my last comment.
        Hide
        cnauroth Chris Nauroth added a comment -

        maybe you can also add the log to the other INTERNAL_SERVER_ERROR clause as well ?

        Arun Suresh, thanks for the review, and good catch. Here is patch v002.

        Show
        cnauroth Chris Nauroth added a comment - maybe you can also add the log to the other INTERNAL_SERVER_ERROR clause as well ? Arun Suresh , thanks for the review, and good catch. Here is patch v002.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 58s trunk passed
        +1 compile 5m 56s trunk passed with JDK v1.8.0_72
        +1 compile 6m 41s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 28s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 18s the patch passed
        +1 compile 5m 39s the patch passed with JDK v1.8.0_72
        +1 javac 5m 39s the patch passed
        +1 compile 6m 36s the patch passed with JDK v1.7.0_95
        +1 javac 6m 36s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 21s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 39s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
        +1 unit 1m 30s hadoop-kms in the patch passed with JDK v1.8.0_72.
        +1 unit 1m 37s hadoop-kms in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        40m 31s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787530/HADOOP-12795.002.patch
        JIRA Issue HADOOP-12795
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d263e2bde355 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 23f937e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8604/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Max memory used 77MB
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8604/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 58s trunk passed +1 compile 5m 56s trunk passed with JDK v1.8.0_72 +1 compile 6m 41s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 18s the patch passed +1 compile 5m 39s the patch passed with JDK v1.8.0_72 +1 javac 5m 39s the patch passed +1 compile 6m 36s the patch passed with JDK v1.7.0_95 +1 javac 6m 36s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 1m 30s hadoop-kms in the patch passed with JDK v1.8.0_72. +1 unit 1m 37s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 40m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787530/HADOOP-12795.002.patch JIRA Issue HADOOP-12795 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d263e2bde355 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23f937e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8604/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8604/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        The patch doesn't appear to include any new or modified tests.

        Once again, it's a logging change only, and I verified it manually.

        I plan to commit this later today, based on Arun's prior "+1 pending one more change".

        Show
        cnauroth Chris Nauroth added a comment - The patch doesn't appear to include any new or modified tests. Once again, it's a logging change only, and I verified it manually. I plan to commit this later today, based on Arun's prior "+1 pending one more change".
        Hide
        cnauroth Chris Nauroth added a comment -

        I have committed this to trunk, branch-2 and branch-2.8. Wei-Chiu Chuang and Arun Suresh, thank you for reviewing.

        Show
        cnauroth Chris Nauroth added a comment - I have committed this to trunk, branch-2 and branch-2.8. Wei-Chiu Chuang and Arun Suresh , thank you for reviewing.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9286 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9286/)
        HADOOP-12795. KMS does not log detailed stack trace for unexpected (cnauroth: rev 70c756d35e6ed5608ce82d1a6fbfb02e19af5ecf)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9286 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9286/ ) HADOOP-12795 . KMS does not log detailed stack trace for unexpected (cnauroth: rev 70c756d35e6ed5608ce82d1a6fbfb02e19af5ecf) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java

          People

          • Assignee:
            cnauroth Chris Nauroth
            Reporter:
            cnauroth Chris Nauroth
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development