[STORM-1537] Upgrade to Kryo 3

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.0
• Epic Link: Release Apache Storm 1.0.0
• Assignee: Abhishek Agarwal
• Reporter: Oscar Boykin
• Created: 10/Feb/16 20:19
• Updated: 25/Mar/16 04:50
• Resolved: 25/Mar/16 04:50

Description

In storm, Kryo (2.21) is used for serialization:
https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L231

The user must use the same version storm does, or there will be a java class error at runtime.

Storm depends on a quasi-abandoned library: carbonite:
https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L210

which depends on Kryo 2.21 and Twitter chill 0.3.6:
https://github.com/sritchie/carbonite/blob/master/project.clj#L8

Chill, currently on 0.7.3, would like to upgrade to Kryo 3.0.3:
https://github.com/twitter/chill/pull/245

because Spark, also depending on chill, would like to upgrade for performance improvements and bugfixes.
https://issues.apache.org/jira/browse/SPARK-11416

Unfortunately, summingbird depends on storm:
https://github.com/twitter/summingbird/blob/develop/build.sbt#L34

so, if chill is upgraded, and that gets on the classpath, summingbird will break at runtime.

I propose:
1) copy the carbonite code into storm. It is likely the only consumer.
2) bump the storm kryo dependency after chill upgrades: recall that storm actually depends on chill-java. A dependency that could possibly be removed after you pull carbonite in.
3) once a new version of storm is published, summingbird (and scalding) can upgrade to the latest chill.

Also, I hope for:
4) we as a JVM community get better about classpath isolation and versioning. Diamonds like this in one big classpath make large codebases very fragile.

Issue Links

• relates to: SPARK-11416 Upgrade kryo package to version 3.0