Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-263

Update Kryo version to 2.21+




      In a nutshell: As reported by Twitter (see below) there are apparently issues with Kryo versions prior to 2.21 which are causing data corruption [1]. Also, albeit of lesser critical importance, is that Storm's current insistence on Kryo 2.17 prevents (or at least unnecessarily complicates) the use of Twitter Chill/Bijection, which are helpful utility libraries to simplify data serialization in Storm topologies (e.g. when using Avro).

      For this reason we may consider upgrading Storm's version of carbonite – a Clojure library for Kryo – which through a transitive dependency determines the actual Kryo version that Storm uses.

        1. Background

      I originally discovered this when I ran into a version conflict between the Kryo versions used by Storm and by Twitter Chill. Storm 0.9.1-incubating (latest version) uses Kryo 2.17 whereas Chill (latest version) uses Kryo 2.21. Without resorting to `exclude` tricks in my build file I couldn't integrate Chill with Storm, and I wanted to use Chill/Bijection to simplify Avro encoding/decoding in my Storm topologies.

      I filed an issue at the Chill project:

      • CHILL-173: Kryo version conflict between Chill and Storm 0.9.1-incubating causes Avro serialization to fail [2]

      Ian O'Connell (@ianoc) replied and pointed out that due to data corruption issues seen in production at Twitter when using Kryo < 2.21 the Chill project cannot downgrade from Kryo 2.21 to Storm's 2.17 version (Scalding, Spark, and Summingbird all use chill with Kryo at 2.21).

      Storm as of 0.9.2 (trunk/master) is currently configured to use carbonite 1.3.2. See the top-level pom.xml.


      Carbonite 1.3.2 depends on Kryo 2.17, where as the recently 1.3.3 depends on Kryo 2.21:

        1. Carbonite*

      Storm uses com.twitter:carbonite, which is maintained by Sam Ritchie (@sritchie) at https://github.com/sritchie/carbonite. Sam would be ok with a patch for carbonite to address this Kryo versioning issue, if needed.

      [1] https://github.com/twitter/chill/issues/173#issuecomment-36534229
      [2] https://github.com/twitter/chill/issues/173


        Issue Links



              ptgoetz P. Taylor Goetz
              miguno Michael G. Noll
              4 Vote for this issue
              11 Start watching this issue