Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-995

Improve Support for YARN 2.2

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.9.0
    • YARN
    • None

    Description

      Due to a nasty conflict relating to the respective protocol buffer libraries in YARN 2.2 and Akka, we had to build a workaround in 0.8.1 for people wanting to build with YARN 2.2 involving different build profiles in Maven and SBT.

      The problem stems from the fact that Hadoop 2.2+ versions pull in a dependency on protobuf 2.5. The akka version (2.0.5) we are currently using depends on protobuf 2.4.1 and they are not binary compatible. This is difficult to solve with classloader isolation because some parts of the code use both akka and the Hadoop libraries simultaneously.

      The solution in the 0.8 branch was to publish our own port of akka 2.0.5 which depends on protobuf 2.5. Then we added a special case in the build where it relies on this special akka version.

      For 0.9 we are pursuing a different solution. We are going to publish or own shaded version of protobuf 2.4 and a version of akka 2.2.4 which relies on that shaded protobof. This will completely isolate the protobuf used in akka vs in other pats of the code. There will no longer be a special case in the build... every build will rely on this special version of akka.

      Attachments

        Activity

          People

            patrick Patrick McFadin
            patrick Patrick McFadin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: