Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      In storm, Kryo (2.21) is used for serialization:
      https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L231

      The user must use the same version storm does, or there will be a java class error at runtime.

      Storm depends on a quasi-abandoned library: carbonite:
      https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L210

      which depends on Kryo 2.21 and Twitter chill 0.3.6:
      https://github.com/sritchie/carbonite/blob/master/project.clj#L8

      Chill, currently on 0.7.3, would like to upgrade to Kryo 3.0.3:
      https://github.com/twitter/chill/pull/245

      because Spark, also depending on chill, would like to upgrade for performance improvements and bugfixes.
      https://issues.apache.org/jira/browse/SPARK-11416

      Unfortunately, summingbird depends on storm:
      https://github.com/twitter/summingbird/blob/develop/build.sbt#L34

      so, if chill is upgraded, and that gets on the classpath, summingbird will break at runtime.

      I propose:
      1) copy the carbonite code into storm. It is likely the only consumer.
      2) bump the storm kryo dependency after chill upgrades: recall that storm actually depends on chill-java. A dependency that could possibly be removed after you pull carbonite in.
      3) once a new version of storm is published, summingbird (and scalding) can upgrade to the latest chill.

      Also, I hope for:
      4) we as a JVM community get better about classpath isolation and versioning. Diamonds like this in one big classpath make large codebases very fragile.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                abhishek.agarwal Abhishek Agarwal
                Reporter:
                posco Oscar Boykin
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: