Details
-
Wish
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
Description
https://github.com/nathanmarz/storm/issues/617
twitter/chill#81
It would be really great to remove the duplication around building Kryo objects (chill-hadoop does it, chill-akka (not yet split out of scala-fish) does it, chill-scala has another one (KryoRegistrationHelper) and storm has one).
It would be really awesome if we (@sritchie, @nathanmarz @jasonjckn @johnynek) could hammer out a plan to have a common dependency.
---------
nathanmarz: I don't view this as very important. It looks like your chill/Storm integration code is about 10 lines, so this doesn't appear to be causing very much pain. Extracting a common dependency sounds good in theory, but as soon as we want to add a new serialization feature to Storm we'll be forking the dependency anyway.
---------
sritchie: The pain is in the diamond dependency on Kryo. Users process the same data in Hadoop and Storm (via Summingbird, for one, but even when writing separate jobs) and serialization versioning mismatches are one of the biggest pains we face.
It's unclear why adding a new serialization feature to storm would need a new fork of Carbonite. That sounds like a design issue.
Attachments
Issue Links
- relates to
-
-
- Resolved
-