[HADOOP-13363] Upgrade protobuf from 2.5.0 to something newer

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha1, 3.0.0-alpha2
• Component/s: build
• Labels: security
• Target Version/s: 3.3.0
• Assignee: Vinayakumar B
• Reporter: Anu Engineer
• Created: 09/Jul/16 00:46
• Updated: 01/Oct/19 12:30

Description

Standard protobuf 2.5.0 does not work properly on many platforms. (See, for example, https://gist.github.com/BennettSmith/7111094 ). In order for us to avoid crazy work arounds in the build environment and the fact that 2.5.0 is starting to slowly disappear as a standard install-able package for even Linux/x86, we need to either upgrade or self bundle or something else.

Issue Links

• is blocked by: HADOOP-13070 classloading isolation improvements for stricter dependencies
• is depended upon by: HADOOP-9991 Fix up Hadoop POMs, roll up JARs to latest versions
• is duplicated by: HDFS-11010 Update to Protobuf 3 in trunk
• is related to: HADOOP-14284 Shade Guava everywhere
• relates to: HBASE-15638 Shade protobuf
• relates to: HDFS-11010 Update to Protobuf 3 in trunk
• relates to: HADOOP-9845 Update protobuf to 2.5 from 2.4.x
• relates to: REEF-1882 Verify whether REEF works with Protocol Buffers 3.x
• links to: GitHub Pull Request #1408

Sub-Tasks

• [pb-upgrade] Update docker image to have 3.7.1 protoc executable: Resolved
• [pb-upgrade] Upgrade protobuf.version to 3.7.1: Resolved
• [pb-upgrade] Update docker image to make 3.7.1 protoc as default: Resolved
• [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes: Resolved
• [HDFS] use protobuf-maven-plugin to generate protobuf classes: Resolved
• [YARN] use protobuf-maven-plugin to generate protobuf classes: Resolved
• [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes: Resolved
• [YARN] Polish the protobuf plugin for hadoop-yarn-csi: In Progress
• [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf: In Progress
• [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency: In Progress
• [pb-upgrade] Remove protocol buffers 3.7.1 from requirements in BUILDING.txt: Resolved
• [pb-upgrade] spark-hive doesn't compile against hadoop trunk because of Token's marshalling: Open