[FLINK-6529] Rework the shading model in Flink

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.1, 1.3.0
• Fix Version/s: 1.4.0
• Component/s: Build System
• Assignee: Chesnay Schepler
• Reporter: Stephan Ewen
• Created: 10/May/17 16:01
• Updated: 02/Oct/19 17:42
• Resolved: 02/Nov/17 15:55

Description

Problem

Currently, Flink shades dependencies like ASM and Guava into all jars of projects that reference it and relocate the classes.

There are some drawbacks to that approach, let's discuss them at the example of ASM:

• The ASM classes are for example in flink-core, flink-java, flink-scala, flink-runtime, etc.

• Users that reference these dependencies have the classes multiple times in the classpath. That is unclean (works, through, because the classes are identical). The same happens when building the final dist. jar.

• Some of these dependencies require to include license files in the shaded jar. It is hard to impossible to build a good automatic solution for that, partly due to Maven's very poor cross-project path support

• Scala does not support shading really well. Scala classes have references to classes in more places than just the class names (apparently for Scala reflect support). Referencing a Scala project with shaded ASM still requires to add a reference to unshaded ASM (at least as a compile dependency).

Proposal

I propose that we build and deploy a asm-flink-shaded version of ASM and directly program against the relocated namespaces. Since we never use classes that we relocate in public interfaces, Flink users will never see the relocated class names. Internally, it does not hurt to use them.

• Proper maven dependency management, no hidden (shaded) dependencies

• one copy of each dependency

• proper Scala interoperability

• no clumsy license management (license is in the deployed asm-flink-shaded)

Issue Links

• is depended upon by: FLINK-6692 The flink-dist jar contains unshaded netty jar

Sub-Tasks

• Add README to "flink-shaded.git" repository: Closed
• Add shaded guava dependency: Closed
• Replace guava dependencies: Closed
• Add shaded netty dependency: Closed
• Replace netty dependencies: Closed
• Add shaded asm dependency: Closed
• Replace asm dependencies: Closed
• Add new import block for shaded dependencies: Closed
• Create flink-shaded-jackson: Closed
• Replace all uses of jackson with flink-shaded-jackson: Closed
• Shade jackson dependency in flink-avro: Closed
• Relocate ZooKeeper: Closed
• Relocate Curator: Closed
• Shade jackson (org.codehouse.jackson) in flink-shaded-hadoop2: Closed