Hadoop Common
  1. Hadoop Common
  2. HADOOP-5103

Too many logs saying "Adding new node" on JobClient console

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      JobClient's console has logs saying "Adding a new node <rackname>/<node-ip-addr>:<port>" for all the hosts, where each split resides.
      For jobs with more #maps, these logs just fill up client's space.

      This is introduced by HADOOP-3293

      1. hadoop-5103.patch
        5 kB
        Jothi Padmanabhan

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #783 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/783/ )
        Hide
        Devaraj Das added a comment -

        I committed this to the 0.20 branch as well.

        Show
        Devaraj Das added a comment - I committed this to the 0.20 branch as well.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #763 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/763/)
        . FileInputFormat now reuses the clusterMap network topology object and that brings down the log messages in the JobClient to do with NetworkTopology.add significantly. Contributed by Jothi Padmanabhan.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #763 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/763/ ) . FileInputFormat now reuses the clusterMap network topology object and that brings down the log messages in the JobClient to do with NetworkTopology.add significantly. Contributed by Jothi Padmanabhan.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Jothi!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Jothi!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12400153/hadoop-5103.patch
        against trunk revision 744224.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3854/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3854/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3854/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3854/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12400153/hadoop-5103.patch against trunk revision 744224. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3854/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3854/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3854/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3854/console This message is automatically generated.
        Hide
        Jothi Padmanabhan added a comment -

        Straight forward fix

        Show
        Jothi Padmanabhan added a comment - Straight forward fix
        Hide
        Jothi Padmanabhan added a comment -

        Yes, that should help. However, that alone would be insufficient. Currently, the getSplits in FileInputFormat calls the getSplitHosts for each file, which in turn uses a new NetworkTopology object – it is a local object in that method. We should use a single clusterMap object across all the Files.

        Show
        Jothi Padmanabhan added a comment - Yes, that should help. However, that alone would be insufficient. Currently, the getSplits in FileInputFormat calls the getSplitHosts for each file, which in turn uses a new NetworkTopology object – it is a local object in that method. We should use a single clusterMap object across all the Files.
        Hide
        Devaraj Das added a comment -

        I just looked at the code of NetworkTopology.add(Node). If we move the LOG.info line inside the block

        if (clusterMap.add(node)) { }

        this problem should be solved, right? I presume that clusterMap.add wouldn't add a node if it is already there and thereby return false for duplicate nodes. We should then see as many LOG messages as the number of successful clusterMap.add calls. The number of such LOG messages should be pretty manageable (and proportional to the number of nodes where the blocks of the splits reside).

        Show
        Devaraj Das added a comment - I just looked at the code of NetworkTopology.add(Node). If we move the LOG.info line inside the block if (clusterMap.add(node)) { } this problem should be solved, right? I presume that clusterMap.add wouldn't add a node if it is already there and thereby return false for duplicate nodes. We should then see as many LOG messages as the number of successful clusterMap.add calls. The number of such LOG messages should be pretty manageable (and proportional to the number of nodes where the blocks of the splits reside).
        Hide
        Hairong Kuang added a comment -

        I hope that the problem could be fixed in JobClient not by changing the logging level proposed above because the "adding..." logging messages in NameNode log have been very useful for us to identify the topology of datanodes.

        Show
        Hairong Kuang added a comment - I hope that the problem could be fixed in JobClient not by changing the logging level proposed above because the "adding..." logging messages in NameNode log have been very useful for us to identify the topology of datanodes.
        Hide
        Jothi Padmanabhan added a comment -

        These statements are from a LOG.info print in NetworkTopology:add. Hadoop-3293 uses NetworkTopology to identify rack/host that have maximum blocks. A simple solution could be to just make these statements LOG.debug instead of LOG.info.

        Show
        Jothi Padmanabhan added a comment - These statements are from a LOG.info print in NetworkTopology:add. Hadoop-3293 uses NetworkTopology to identify rack/host that have maximum blocks. A simple solution could be to just make these statements LOG.debug instead of LOG.info.

          People

          • Assignee:
            Jothi Padmanabhan
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development