Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-6369

2.11 Spark integration has a dependency on Spark's 2.10 library

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: 2.4
    • Component/s: spark
    • Labels:
      None

      Description

      A simple Spark application that uses both Spark and Ignite fails to compile in sbt under 2.11 due to conflicting dependencies. For a simple sbt definition with:

      libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0" 
      libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0" 
      libraryDependencies += "org.apache.ignite" % "ignite-spark" % "2.1.0"
      

      it will fail to compile with:

      [error] Modules were resolved with conflicting cross-version suffixes in: 
      [error]    org.scalatest:scalatest _2.11, _2.10 
      [error]    com.twitter:chill _2.11, _2.10 
      [error]    org.apache.spark:spark-unsafe _2.11, _2.10 
      [error]    org.apache.spark:spark-tags _2.11, _2.10 
      [trace] Stack trace suppressed: run 'last *:update' for the full output. 
      

      It looks like a single culprit is the entry in ignite-spark's pom.xml for
      spark-unsafe_2.10.
      When removed, compiled and published, everything works great.
      I don't see why such an entry exists in spark module when there is a separate spark-2.10 module as well.

      Happy to submit a PR if anyone is willing to give a thumbs up.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hubertp Hubert Plociniczak
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: