Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14515

Specifically configure zookeeper-related log levels in KMS log4j

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: kms
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When investigating a case, we tried to turn on KMS DEBUG by setting the root logger in the log4j to DEBUG. This ends up making org.apache.zookeeper.ClientCnxn to generate 199.2M out of a 200M log file, which made the kms.log rotate very quickly.

      We should keep zookeeper's log unaffected by the root logger, and only turn it on when interested.

      1. HADOOP-14515.01.patch
        0.9 kB
        Xiao Chen
      2. HADOOP-14515.addendum.patch
        0.8 kB
        Xiao Chen

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11939 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11939/)
        HADOOP-14515. Addendum. Specifically configure zookeeper-related log (xiao: rev fb5ee3fafbe5a7390aa706cdeef12732051b58ee)

        • (edit) hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11939 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11939/ ) HADOOP-14515 . Addendum. Specifically configure zookeeper-related log (xiao: rev fb5ee3fafbe5a7390aa706cdeef12732051b58ee) (edit) hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
        Hide
        xiaochen Xiao Chen added a comment -

        Committed the addendum patch to trunk, branch-2 and branch-2.8.
        Thanks.

        Show
        xiaochen Xiao Chen added a comment - Committed the addendum patch to trunk, branch-2 and branch-2.8. Thanks.
        Hide
        xiaochen Xiao Chen added a comment -

        Log level changes so no test added.
        Thanks again, Yongjun. Committing this..

        Show
        xiaochen Xiao Chen added a comment - Log level changes so no test added. Thanks again, Yongjun. Committing this..
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 55s trunk passed
        +1 mvnsite 0m 42s trunk passed
        +1 mvnsite 0m 39s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 unit 0m 14s hadoop-kms in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        16m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14515
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874802/HADOOP-14515.addendum.patch
        Optional Tests asflicense mvnsite unit
        uname Linux 8c94df1af6c4 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a5c0476
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12643/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12643/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 55s trunk passed +1 mvnsite 0m 42s trunk passed +1 mvnsite 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 unit 0m 14s hadoop-kms in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 16m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14515 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874802/HADOOP-14515.addendum.patch Optional Tests asflicense mvnsite unit uname Linux 8c94df1af6c4 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a5c0476 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12643/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12643/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Xiao Chen. +1 on the addendum patch.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Xiao Chen . +1 on the addendum patch.
        Hide
        xiaochen Xiao Chen added a comment -

        It seems I missed the main log4j here.... attaching addendum patch.

        Show
        xiaochen Xiao Chen added a comment - It seems I missed the main log4j here.... attaching addendum patch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11891 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11891/)
        HADOOP-14515. Specifically configure zookeeper-related log levels in KMS (xiao: rev 15964ef1473726ef0f29949eda00932d19c64635)

        • (edit) hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11891 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11891/ ) HADOOP-14515 . Specifically configure zookeeper-related log levels in KMS (xiao: rev 15964ef1473726ef0f29949eda00932d19c64635) (edit) hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties
        Hide
        xiaochen Xiao Chen added a comment -

        Committed to trunk, branch-2 and branch-2.8.
        Thanks Yongjun Zhang for the review.

        Show
        xiaochen Xiao Chen added a comment - Committed to trunk, branch-2 and branch-2.8. Thanks Yongjun Zhang for the review.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Xiao Chen, +1 on the patch.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Xiao Chen , +1 on the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 15m 29s trunk passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 unit 0m 16s hadoop-kms in the patch passed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        16m 53s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14515
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872350/HADOOP-14515.01.patch
        Optional Tests asflicense unit
        uname Linux 79f03f92c43a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5578af8
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12508/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12508/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 29s trunk passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 unit 0m 16s hadoop-kms in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 16m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14515 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872350/HADOOP-14515.01.patch Optional Tests asflicense unit uname Linux 79f03f92c43a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5578af8 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12508/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12508/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            xiaochen Xiao Chen
            Reporter:
            xiaochen Xiao Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development