Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3350

YARN RackResolver spams logs with messages at info level

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When you run an application the container logs shows a lot of messages for the RackResolver:

      2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15

      A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space.

      1. YARN-3350.2.patch
        1 kB
        Wilfred Spiegelenburg
      2. YARN-3350.patch
        1 kB
        Wilfred Spiegelenburg
      3. yarn-RackResolver-log.txt
        3 kB
        Wilfred Spiegelenburg

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #139 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/139/)
        YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #139 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/139/ ) YARN-3350 . YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #130 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/130/)
        YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #130 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/130/ ) YARN-3350 . YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2071 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2071/)
        YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2071 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2071/ ) YARN-3350 . YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2089 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2089/)
        YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2089 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2089/ ) YARN-3350 . YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #873 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/873/)
        YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #873 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/873/ ) YARN-3350 . YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #139 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/139/)
        YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #139 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/139/ ) YARN-3350 . YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7391 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7391/)
        YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7391 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7391/ ) YARN-3350 . YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
        Hide
        djp Junping Du added a comment -

        I have commit v2 patch to trunk and branch-2. Thanks Wilfred Spiegelenburg for contributing the patch!

        Show
        djp Junping Du added a comment - I have commit v2 patch to trunk and branch-2. Thanks Wilfred Spiegelenburg for contributing the patch!
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12704841/YARN-3350.2.patch
        against trunk revision ed4e72a.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6977//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6977//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704841/YARN-3350.2.patch against trunk revision ed4e72a. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6977//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6977//console This message is automatically generated.
        Hide
        djp Junping Du added a comment -

        Thanks Wilfred Spiegelenburg for updating the patch. +1 on latest patch.
        Will commit it tomorrow if no further comments from others.

        Show
        djp Junping Du added a comment - Thanks Wilfred Spiegelenburg for updating the patch. +1 on latest patch. Will commit it tomorrow if no further comments from others.
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        updated patch wrapping the log in a if debug

        Show
        wilfreds Wilfred Spiegelenburg added a comment - updated patch wrapping the log in a if debug
        Hide
        djp Junping Du added a comment -

        Just get confirmation from Sandy at SPARK-5393. Wilfred Spiegelenburg, mind go ahead to deliver an updated patch here? Thanks!

        Show
        djp Junping Du added a comment - Just get confirmation from Sandy at SPARK-5393 . Wilfred Spiegelenburg , mind go ahead to deliver an updated patch here? Thanks!
        Hide
        djp Junping Du added a comment -

        Sorry for typo: is LOG is not enabling debug level => if LOG is not enabling debug level

        Show
        djp Junping Du added a comment - Sorry for typo: is LOG is not enabling debug level => if LOG is not enabling debug level
        Hide
        djp Junping Du added a comment -

        Thanks Wilfred Spiegelenburg for reporting the issue and delivering a quick patch to fix it.
        A nit comment for the patch: LOG.debug() can be wrap inside of "if (LOG.isDebugEnabled())" which could be faster is LOG is not enabling debug level.
        I just pinged Sandy Ryza on SPARK-5393 as I am curious that why we didn't fix it in YARN side earlier. Let's wait for his feedback or we may miss something here.

        Show
        djp Junping Du added a comment - Thanks Wilfred Spiegelenburg for reporting the issue and delivering a quick patch to fix it. A nit comment for the patch: LOG.debug() can be wrap inside of "if (LOG.isDebugEnabled())" which could be faster is LOG is not enabling debug level. I just pinged Sandy Ryza on SPARK-5393 as I am curious that why we didn't fix it in YARN side earlier. Let's wait for his feedback or we may miss something here.
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        No tests for this change: it is just a simple log level change without any further code changes.

        BTW: this same issue was encountered by Spark as an application using the RackResolver and they changed the log level from their side to prevent log flooding via SPARK-5393

        Show
        wilfreds Wilfred Spiegelenburg added a comment - No tests for this change: it is just a simple log level change without any further code changes. BTW: this same issue was encountered by Spark as an application using the RackResolver and they changed the log level from their side to prevent log flooding via SPARK-5393
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12704712/YARN-3350.patch
        against trunk revision 3ff1ba2.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6970//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6970//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704712/YARN-3350.patch against trunk revision 3ff1ba2. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6970//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6970//console This message is automatically generated.
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        Patch to change the log level to debug for the messages. The messages are not relevant for tracking and should not be logged during the normal operation of the application.

        Show
        wilfreds Wilfred Spiegelenburg added a comment - Patch to change the log level to debug for the messages. The messages are not relevant for tracking and should not be logged during the normal operation of the application.
        Hide
        wilfreds Wilfred Spiegelenburg added a comment -

        Extract from a log which shows the messages logged by the RackResolver

        Show
        wilfreds Wilfred Spiegelenburg added a comment - Extract from a log which shows the messages logged by the RackResolver

          People

          • Assignee:
            wilfreds Wilfred Spiegelenburg
            Reporter:
            wilfreds Wilfred Spiegelenburg
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development