[CARBONDATA-3389] Optimize the bundling of Hadoop libraries with CarbonData

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.3
• Component/s: hadoop-integration
• Assignee: Unassigned
• Reporter: lamber-ken
• Created: 16/May/19 03:35
• Updated: 02/Jun/20 03:51

Description

For now, CarbonData provides archives bunding with hadoop-2.7.2, user needs to build carbondata to fit their own hadoop env.

apache-carbondata-1.5.3-bin-spark2.1.0-hadoop2.7.2.jar
apache-carbondata-1.5.3-bin-spark2.1.0-hadoop2.7.2.jar.asc
apache-carbondata-1.5.3-bin-spark2.1.0-hadoop2.7.2.jar.sha512
apache-carbondata-1.5.3-bin-spark2.2.1-hadoop2.7.2.jar
apache-carbondata-1.5.3-bin-spark2.2.1-hadoop2.7.2.jar.asc
apache-carbondata-1.5.3-bin-spark2.2.1-hadoop2.7.2.jar.sha512
apache-carbondata-1.5.3-bin-spark2.3.2-hadoop2.7.2.jar
apache-carbondata-1.5.3-bin-spark2.3.2-hadoop2.7.2.jar.asc
apache-carbondata-1.5.3-bin-spark2.3.2-hadoop2.7.2.jar.sha512

 

I think it's better to split carbondata and hadoop. use can manually download a pre-packaged Hadoop jar from the optional components, like bellow

CarbonData 1.6.0
CarbonData 1.6.0 for Scala 2.11
CarbonData 1.6.0 for Scala 2.12

Optional components
Pre-bundled Hadoop 2.4.1
Pre-bundled Hadoop 2.6.5
Pre-bundled Hadoop 2.7.5
Pre-bundled Hadoop 2.8.3