[BIGTOP-3605] Define Bigtop 3.1 release BOM

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Assignee: Masatake Iwasaki
• Reporter: Masatake Iwasaki
• Created: 22/Nov/21 04:03
• Updated: 01/May/22 23:42
• Resolved: 25/Apr/22 15:40

Description

It's about time to discuss the next release BOM

My proposal here is releasing Bigtop 3.1 shortly based on Hadoop 3.2 before bumping Hadoop to 3.3 (or later).

• Bumping to Hadoop 3.3 is big change. Providing Hadoop 3.2.3 containing almost 300 fixes from 3.2.2 could be useful for users of Bigtop 3.0.
• HBase 2.2.6 and Kafka 2.4.1 are bit old releases. We can upgrade them by bumping ZooKeeper to 3.5.

Components

Components               in v3.0   in v3.1
alluxio                   2.4.1     2.4.1
ambari                    2.7.5     2.7.5
bigtop-ambari-mpack       2.7.5     2.7.5
bigtop_groovy             2.5.4     2.5.4  
bigtop_jsvc               1.0.15 => 1.0.15 or 1.2.3 [1]
bigtop_utils              3.0.0
elasticsearch             5.6.14    5.6.14 (or greater, as far as its license is ASLv2-compliant)
flink                     1.11.3 => 1.11.6
gpdb                      5.28.5
hadoop                    3.2.2  => 3.2.3
hbase                     2.2.6  => 2.4.11
hive                      3.1.2     3.1.2
kafka                     2.4.1  => 2.8.1
kibana                    5.4.1     5.4.1 (or greater, as far as its license is ASLv2-compliant)
livy                      0.7.1     0.7.1
logstash                  5.4.1     5.4.1 (or greater, as far as its license is ASLv2-compliant)
oozie                     5.2.1     5.2.1
phoenix                   5.1.0  => 5.1.2
solr                      8.7.0  => 8.11.1
spark                     3.0.1  => 3.1.2 [2]
sqoop                     1.4.7     1.4.7
tez                       0.10.0 => 0.10.1
ycsb                      0.17.0    0.17.0
zeppelin                  0.9.0  => 0.10.0 
zookeeper                 3.4.14 => 3.5.9

Distros

- CentOS 7, Rocky Linux 8 (replacing CentOS 8)
- Debian 10, 11 (dropping 9 and adding 11)
- Fedora 35 (upgraded from 33)
- Ubuntu 18.04, 20.04

Archs

- x86_64
- aarch64
- ppc64le

JDK

JDK8

[1]: https://issues.apache.org/jira/browse/BIGTOP-3604
[2]: https://github.com/apache/zeppelin/blob/v0.10.0/spark/spark-shims/src/main/scala/org/apache/zeppelin/spark/SparkVersion.java#L39

Issue Links

• is related to: BIGTOP-3600 Add support for Debian 11 Bullseye
• is related to: BIGTOP-3637 Add Rocky Linux 8 support
• is related to: BIGTOP-3629 Drop Debian 9 support - second attempt
• is related to: BIGTOP-3658 Add Fedora 35 support and drop 33

Sub-Tasks

• Bump ZooKeeper to 3.5.9: Resolved
• Fix build failure of Hadoop 3.2 against ZooKeeper 3.5: Resolved
• Bump HBase to 2.4.8: Resolved
• Bump Phoenix to 5.1.2: Resolved
• Bump Spark to 3.1.2: Resolved
• Bump Zeppelin to 0.10.0: Resolved
• Bump Kafka to 2.8.1: Resolved
• Bump Hadoop to 3.2.3: Resolved
• Use a stricter match for the zookeeper jar in hadoop packaging: Resolved
• Review log4j configurations for CVE-2021-44228: Resolved
• Bump HBase to 2.4.11: Resolved