Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14956

Spark dependencies conflicts

    XMLWordPrintableJSON

    Details

    • Type: Brainstorming
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: 1.6.1
    • Fix Version/s: None
    • Component/s: Spark Core

      Description

      Since Spark lives with ages-old dependencies, it's all too often a problem that we must downgrade one of our dependencies just to make it "not explode" in Spark. And most importantly, this is something we only encounter at runtime and that makes it an even worse problem.

      So the usual solution when we depend directly on a package (but not one of our transitive dependencies) is to relocate with a shade plugin like this:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
      ...
        <configuration>
          <relocations>
            <relocation>
              <pattern>scopt</pattern>
              <shadedPattern>hidden.scopt</shadedPattern>
            </relocation>
      

      Other times, we must exclude transitive dependencies like that:

              <dependency>
                  <groupId>com.twitter.penguin</groupId>
                  <artifactId>korean-text</artifactId>
                  <version>4.1.2</version>
                  <exclusions>
                      <exclusion>
                          <groupId>org.slf4j</groupId>
                          <artifactId>slf4j-nop</artifactId>
                      </exclusion>
                  </exclusions>
              </dependency>
      

      Everything related to Guava, Log4j, Databind, scopt and even Java8 fall into this category.

      I wonder if that would be possible to use OSGi style of plugins when running code inside Spark.. That would shield us from all theses things.

      Could bill-of-material be of any help here as well?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                FlamingMike Michel Lemay
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: