Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12281

Add an incompatible version of Kryo to the hadoop-client classpath

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change

      Description

      Although versions of guava & jackson create compatibility problems, the issue of Kryo compatibility is put off until Hive, Spark and other libraries are loaded on the same classpath. By adding a version of Kryo to the Hadoop core libraries, we can break everything downstream in a consistent manner.

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          resolving as a wontfix in case more people start taking it seriously.

          Show
          stevel@apache.org Steve Loughran added a comment - resolving as a wontfix in case more people start taking it seriously.
          Hide
          cmccabe Colin P. McCabe added a comment -

          You're not on my junk email filter, Steve Loughran. I just have too many emails Anyway, this JIRA made me smile, so it can't be all bad.

          Show
          cmccabe Colin P. McCabe added a comment - You're not on my junk email filter, Steve Loughran . I just have too many emails Anyway, this JIRA made me smile, so it can't be all bad.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I sent you a private email to avoid public embarrassment. I must be on your junk email sender filter...

          Show
          stevel@apache.org Steve Loughran added a comment - I sent you a private email to avoid public embarrassment. I must be on your junk email sender filter...
          Hide
          cmccabe Colin P. McCabe added a comment -

          Well, you got me. It was a massive troll all along.

          Show
          cmccabe Colin P. McCabe added a comment - Well, you got me. It was a massive troll all along.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Its worthwhile noting that this JIRA wasn't meant to be taken seriously. Hence the title. It's related to SPARK-8064 and the effort required to get Hive and Spark sharing the same Kryo version.

          Show
          stevel@apache.org Steve Loughran added a comment - Its worthwhile noting that this JIRA wasn't meant to be taken seriously. Hence the title. It's related to SPARK-8064 and the effort required to get Hive and Spark sharing the same Kryo version.
          Hide
          busbey Sean Busbey added a comment -

          Isn't the place to handle integration of Hadoop and e.g. Spark, Hive, or whatever within the BigTop project? Why are we making ourselves antagonistic wrt the rest of the ecosystem?

          Show
          busbey Sean Busbey added a comment - Isn't the place to handle integration of Hadoop and e.g. Spark, Hive, or whatever within the BigTop project? Why are we making ourselves antagonistic wrt the rest of the ecosystem?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          By adding a version of kryo to the base classpath we can know from the outset that everything which uses kryo won't work, rather than waiting for this to show up at integration test time.

          Show
          stevel@apache.org Steve Loughran added a comment - By adding a version of kryo to the base classpath we can know from the outset that everything which uses kryo won't work, rather than waiting for this to show up at integration test time.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Hadoop doesn't use Kryo, I don't see the motivation here. What am I missing?

          Show
          cmccabe Colin P. McCabe added a comment - Hadoop doesn't use Kryo, I don't see the motivation here. What am I missing?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          note there's a risk that we actually commit a version which is compatible with a dependent project. We can avoid this by having classes with the same name as the kryo ones, but different signatures & behaviour. That way incompatibility is guarantees.

          Show
          stevel@apache.org Steve Loughran added a comment - note there's a risk that we actually commit a version which is compatible with a dependent project. We can avoid this by having classes with the same name as the kryo ones, but different signatures & behaviour. That way incompatibility is guarantees.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Gopal V do you want to take a crack at this?

          Show
          hagleitn Gunther Hagleitner added a comment - Gopal V do you want to take a crack at this?

            People

            • Assignee:
              hagleitn Gunther Hagleitner
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development