Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3157

kudu-spark2 jar is now including slf4j classes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.12.0
    • Fix Version/s: 1.14.0
    • Component/s: spark
    • Labels:
      None

      Description

      It seems that sometime between versions 1.7.0 and 1.12.0 of org.apache.kudu:kudu-spark2_2.11, some SLF4J classes started being included.

      # for 1.7.0
      jar tvf  ~/.m2/repository/org/apache/kudu/kudu-spark2_2.11/1.7.0/kudu-spark2_2.11-1.7.0.jar | grep slf4j
        1203 Wed Jun 29 14:43:04 CDT 2016 META-INF/license/LICENSE.slf4j.txt
      #for 1.12.0
       jar tvf  ~/.m2/repository/org/apache/kudu/kudu-spark2_2.11/1.12.0/kudu-spark2_2.11-1.12.0.jar | grep slf4j
           0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/
           0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/event/
         555 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/EventConstants.class
        9842 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/EventRecodingLogger.class
        1450 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/Level.class
         432 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/LoggingEvent.class
        2615 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/SubstituteLoggingEvent.class
           0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/helpers/
        3285 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMarker.class
        1634 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMarkerFactory.class
        1233 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMDCAdapter$1.class
        2618 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMDCAdapter.class
        1091 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/FormattingTuple.class
        4773 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/MarkerIgnoringBase.class
        7553 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/MessageFormatter.class
         823 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NamedLoggerBase.class
        3267 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPLogger.class
         584 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPLoggerFactory.class
        1249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPMDCAdapter.class
       10262 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/SubstituteLogger.class
        2199 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/SubstituteLoggerFactory.class
         196 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/Util$1.class
         765 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/Util$ClassContextSecurityManager.class
        2952 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/Util.class
         180 Mon Dec 16 22:03:32 CST 2019 org/slf4j/ILoggerFactory.class
         272 Mon Dec 16 22:03:32 CST 2019 org/slf4j/IMarkerFactory.class
        1375 Mon Dec 16 22:03:32 CST 2019 org/slf4j/Logger.class
       12575 Mon Dec 16 22:03:32 CST 2019 org/slf4j/LoggerFactory.class
         666 Mon Dec 16 22:03:32 CST 2019 org/slf4j/Marker.class
        1621 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MarkerFactory.class
         177 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC$1.class
         749 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC$MDCCloseable.class
        3675 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC.class
           0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/spi/
         455 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/LocationAwareLogger.class
         249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/LoggerFactoryBinder.class
         249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/MarkerFactoryBinder.class
         529 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/MDCAdapter.class
           0 Mon Apr 27 14:53:30 CDT 2020 META-INF/maven/org.slf4j/
           0 Mon Apr 27 14:53:30 CDT 2020 META-INF/maven/org.slf4j/slf4j-api/
        3835 Mon Dec 16 21:59:52 CST 2019 META-INF/maven/org.slf4j/slf4j-api/pom.xml
         109 Mon Dec 16 22:00:08 CST 2019 META-INF/maven/org.slf4j/slf4j-api/pom.properties
      

      These should be removed from the jar, or at least relocated so as not to clash with Spark vendor dependencies, which can cause LinkageError in Spark clusters. Real life example of this problem, seen on a Spark job started on CDH 6.3 cluster:

      Exception in thread "main" java.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of org/apache/spark/util/ChildFirstURLClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method\'s defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature
              at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:423)
              at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
              at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
      

        Attachments

          Activity

            People

            • Assignee:
              granthenke Grant Henke
              Reporter:
              jeff.w.evans Jeff Evans
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: