Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14369

NetworkTopology calls expensive toString() when logging

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None

      Description

      When NetworkTopology calls add() or remove(), it calls toString() for LOG.debug() which is pretty expensive.

      1. HADOOP-14369-000.patch
        2 kB
        Íñigo Goiri
      2. HADOOP-14369-001.patch
        2 kB
        Íñigo Goiri

        Issue Links

          Activity

          Hide
          elgoiri Íñigo Goiri added a comment -

          Removing toString() from LOG.debug() messages.

          Show
          elgoiri Íñigo Goiri added a comment - Removing toString() from LOG.debug() messages.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Xiao Chen, this was introduced in HDFS-10320 when switching to Logger.
          I think this is the right way to avoid building the whole String.
          I think we should also fix:

          LOG.debug("Failed to find datanode (scope=\"{}\" excludedScope=\"{}\").",
              String.valueOf(scope), String.valueOf(excludedScope));
          

          Not sure of the reason for valueOf().

          Show
          elgoiri Íñigo Goiri added a comment - Xiao Chen , this was introduced in HDFS-10320 when switching to Logger . I think this is the right way to avoid building the whole String. I think we should also fix: LOG.debug( "Failed to find datanode (scope=\" {}\ " excludedScope=\" {}\ ")." , String .valueOf(scope), String .valueOf(excludedScope)); Not sure of the reason for valueOf() .
          Hide
          andrew.wang Andrew Wang added a comment -

          Xiao's on leave right now, but I can review. 000 patch LGTM, let's see precommit.

          Show
          andrew.wang Andrew Wang added a comment - Xiao's on leave right now, but I can review. 000 patch LGTM, let's see precommit.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 39s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 51s trunk passed
          +1 compile 20m 4s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 21s trunk passed
          -1 findbugs 1m 30s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 19m 26s the patch passed
          +1 javac 19m 26s the patch passed
          +1 checkstyle 0m 41s the patch passed
          +1 mvnsite 1m 15s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 51s the patch passed
          -1 unit 9m 1s hadoop-common in the patch failed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          90m 20s



          Reason Tests
          Failed junit tests hadoop.security.TestRaceWhenRelogin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14369
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865839/HADOOP-14369-000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a16f205ba0f6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1058b40
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12219/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12219/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12219/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12219/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 39s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 51s trunk passed +1 compile 20m 4s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 21s trunk passed -1 findbugs 1m 30s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 19m 26s the patch passed +1 javac 19m 26s the patch passed +1 checkstyle 0m 41s the patch passed +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 51s the patch passed -1 unit 9m 1s hadoop-common in the patch failed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 90m 20s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14369 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865839/HADOOP-14369-000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a16f205ba0f6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1058b40 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12219/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12219/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12219/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12219/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Removing unnecessary String#valueOf() from LOG.debug()

          Show
          elgoiri Íñigo Goiri added a comment - Removing unnecessary String#valueOf() from LOG.debug()
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 36s trunk passed
          +1 compile 16m 39s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          -1 findbugs 1m 23s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 0m 39s the patch passed
          +1 compile 13m 59s the patch passed
          +1 javac 13m 59s the patch passed
          +1 checkstyle 0m 35s the patch passed
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 30s the patch passed
          +1 javadoc 0m 49s the patch passed
          +1 unit 7m 56s hadoop-common in the patch passed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          64m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14369
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865863/HADOOP-14369-001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 654cf138a45e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6bf42e4
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12220/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12220/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12220/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 36s trunk passed +1 compile 16m 39s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 20s trunk passed -1 findbugs 1m 23s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 13m 59s the patch passed +1 javac 13m 59s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 7m 56s hadoop-common in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 64m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14369 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865863/HADOOP-14369-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 654cf138a45e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6bf42e4 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12220/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12220/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12220/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          The FindBugs are unrelated (I've been seeing this weird behavior in other JIRAs too) and I don't think there is a unit test to add here.

          Show
          elgoiri Íñigo Goiri added a comment - The FindBugs are unrelated (I've been seeing this weird behavior in other JIRAs too) and I don't think there is a unit test to add here.
          Hide
          andrew.wang Andrew Wang added a comment -

          +1 LGTM. For future reference, "extant" means that these findbugs warnings already exist. We upgraded from findbugs to spotbugs, and are still in the process of cleaning up the warnings.

          Thanks for the contribution!

          Show
          andrew.wang Andrew Wang added a comment - +1 LGTM. For future reference, "extant" means that these findbugs warnings already exist. We upgraded from findbugs to spotbugs, and are still in the process of cleaning up the warnings. Thanks for the contribution!
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, branch-2, branch-2.8. Thanks again for the contribution Inigo!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, branch-2, branch-2.8. Thanks again for the contribution Inigo!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11672 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11672/)
          HADOOP-14369. NetworkTopology calls expensive toString() when logging. (wang: rev dcc292d7ded200a4976f6d348952ecba10f01db2)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11672 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11672/ ) HADOOP-14369 . NetworkTopology calls expensive toString() when logging. (wang: rev dcc292d7ded200a4976f6d348952ecba10f01db2) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          Hide
          xiaochen Xiao Chen added a comment -

          Good catch Íñigo Goiri, thanks for the fix. Also thanks Andrew for the review+commit.

          Show
          xiaochen Xiao Chen added a comment - Good catch Íñigo Goiri , thanks for the fix. Also thanks Andrew for the review+commit.

            People

            • Assignee:
              elgoiri Íñigo Goiri
              Reporter:
              elgoiri Íñigo Goiri
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development