[STORM-263] Update Kryo version to 2.21+

• Type: Dependency upgrade
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2-incubating
• Fix Version/s: 0.9.2-incubating
• Component/s: storm-core
• Labels: carbonite kryo serialization
• Assignee: P. Taylor Goetz
• Reporter: Michael G. Noll
• Created: 24/Mar/14 09:32
• Updated: 05/Oct/15 02:53
• Resolved: 04/Apr/14 18:42

Description

In a nutshell: As reported by Twitter (see below) there are apparently issues with Kryo versions prior to 2.21 which are causing data corruption [1]. Also, albeit of lesser critical importance, is that Storm's current insistence on Kryo 2.17 prevents (or at least unnecessarily complicates) the use of Twitter Chill/Bijection, which are helpful utility libraries to simplify data serialization in Storm topologies (e.g. when using Avro).

For this reason we may consider upgrading Storm's version of carbonite – a Clojure library for Kryo – which through a transitive dependency determines the actual Kryo version that Storm uses.

1. 1. Background

I originally discovered this when I ran into a version conflict between the Kryo versions used by Storm and by Twitter Chill. Storm 0.9.1-incubating (latest version) uses Kryo 2.17 whereas Chill (latest version) uses Kryo 2.21. Without resorting to `exclude` tricks in my build file I couldn't integrate Chill with Storm, and I wanted to use Chill/Bijection to simplify Avro encoding/decoding in my Storm topologies.

I filed an issue at the Chill project:

• CHILL-173: Kryo version conflict between Chill and Storm 0.9.1-incubating causes Avro serialization to fail [2]

Ian O'Connell (@ianoc) replied and pointed out that due to data corruption issues seen in production at Twitter when using Kryo < 2.21 the Chill project cannot downgrade from Kryo 2.21 to Storm's 2.17 version (Scalding, Spark, and Summingbird all use chill with Kryo at 2.21).

Storm as of 0.9.2 (trunk/master) is currently configured to use carbonite 1.3.2. See the top-level pom.xml.

<carbonite.version>1.3.2</carbonite.version>

Carbonite 1.3.2 depends on Kryo 2.17, where as the recently 1.3.3 depends on Kryo 2.21:
https://github.com/sritchie/carbonite/blob/master/project.clj

1. 1. Carbonite*

Storm uses com.twitter:carbonite, which is maintained by Sam Ritchie (@sritchie) at https://github.com/sritchie/carbonite. Sam would be ok with a patch for carbonite to address this Kryo versioning issue, if needed.

[1] https://github.com/twitter/chill/issues/173#issuecomment-36534229
[2] https://github.com/twitter/chill/issues/173

Issue Links

• is related to: STORM-161 Proposal: move the Kryo config code out to chill-java