Hadoop Common
  1. Hadoop Common
  2. HADOOP-8005

Multiple SLF4J binding message in .out file for all daemons

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: scripts
    • Labels:
      None

      Description

      When I start the NameNode or DataNode using sbin/hadoop-daemon.sh, I get a variant of the following error on stdout:

      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/Users/joecrow/Code/hadoop-0.23.0/share/hadoop/common/lib/slf4j-log4j12-1.5.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/Users/joecrow/Code/hadoop-0.23.0/share/hadoop/hdfs/lib/slf4j-log4j12-1.5.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      

        Issue Links

          Activity

          Hide
          Joe Crobak added a comment -

          Could change the scope of slf4j-log4j12 to "provided" for the hdfs project so that jar is not included in the hdfs project's lib directory.

          Show
          Joe Crobak added a comment - Could change the scope of slf4j-log4j12 to "provided" for the hdfs project so that jar is not included in the hdfs project's lib directory.
          Hide
          Harsh J added a comment -

          Joe,

          That makes senes. Would you be filing a patch changing that in the maven conf.?

          Show
          Harsh J added a comment - Joe, That makes senes. Would you be filing a patch changing that in the maven conf.?
          Hide
          Joe Crobak added a comment -

          Will do!

          Show
          Joe Crobak added a comment - Will do!
          Hide
          Harsh J added a comment -

          Oops, s/senes/sense

          Show
          Harsh J added a comment - Oops, s/senes/sense
          Hide
          Harsh J added a comment -

          Moved to HADOOP as it targets all daemons. Edited summary to match that.

          Show
          Harsh J added a comment - Moved to HADOOP as it targets all daemons. Edited summary to match that.
          Hide
          Eli Collins added a comment -

          Would be good to fix this for 23.1, makes the start-* scripts much noisier.

          Show
          Eli Collins added a comment - Would be good to fix this for 23.1, makes the start-* scripts much noisier.
          Hide
          Joe Crobak added a comment -

          Should we change the slf4j-log4j12 dependency to "provided" in hadoop-common, then, rather than trying to sprinkle it throughout the other projects? Or is that too dangerous/invasive?

          Sorry for taking so long to get a patch – I'll try to get to it tonight, but I won't be offended if someone else has the time and does it.

          Show
          Joe Crobak added a comment - Should we change the slf4j-log4j12 dependency to "provided" in hadoop-common, then, rather than trying to sprinkle it throughout the other projects? Or is that too dangerous/invasive? Sorry for taking so long to get a patch – I'll try to get to it tonight, but I won't be offended if someone else has the time and does it.
          Hide
          Joe Crobak added a comment -

          So I tried changing slf4j-log4j12 to "provided" in hadoop-common, but this didn't fix the problem. It turns out that the startup scripts add all of the libs in both the hdfs and mapred dirs to the classpath for all daemons, so it appeared on the classpath, twice.

          It seems like the best solution, for the startup scripts, at least, is to exclude the jars from hdfs/mapred and then just keep the jar in hadoop-common. This is contrary to the best-practices for slf4j when your jar is a "library" jar though – in which case an adapter shouldn't be included.

          Thoughts?

          Show
          Joe Crobak added a comment - So I tried changing slf4j-log4j12 to "provided" in hadoop-common, but this didn't fix the problem. It turns out that the startup scripts add all of the libs in both the hdfs and mapred dirs to the classpath for all daemons, so it appeared on the classpath, twice. It seems like the best solution, for the startup scripts, at least, is to exclude the jars from hdfs/mapred and then just keep the jar in hadoop-common. This is contrary to the best-practices for slf4j when your jar is a "library" jar though – in which case an adapter shouldn't be included. Thoughts?
          Hide
          Jason Lowe added a comment -

          We can also address the problem by removing the slf4j jars from the hdfs and mapreduce assemblies. HDFS no longer seems to need slf4j, and we can explicitly filter the slf4j jars from the mapreduce project. Patch attached for comment.

          If desired, I can file the HDFS and MAPREDUCE JIRAs to get the corresponding changes into those projects.

          Show
          Jason Lowe added a comment - We can also address the problem by removing the slf4j jars from the hdfs and mapreduce assemblies. HDFS no longer seems to need slf4j, and we can explicitly filter the slf4j jars from the mapreduce project. Patch attached for comment. If desired, I can file the HDFS and MAPREDUCE JIRAs to get the corresponding changes into those projects.
          Hide
          Jason Lowe added a comment -

          Resolving since this is fixed by the combination of HDFS-3136 and MAPREDUCE-4060, both of which were committed to branch-2 and trunk.

          Show
          Jason Lowe added a comment - Resolving since this is fixed by the combination of HDFS-3136 and MAPREDUCE-4060 , both of which were committed to branch-2 and trunk.
          Hide
          Jason Lowe added a comment -

          Updating fixed versions, as the two fixes have now been merged to 0.23.

          Show
          Jason Lowe added a comment - Updating fixed versions, as the two fixes have now been merged to 0.23.

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Joe Crobak
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development