Details
-
Task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
It's about time to discuss the next release BOM 
Based on the past discussions [1][2], we're going to focus "core" components in the next release and drop the ones (at least temporarily) which are
- not so widely used by our users
- not actively developed recently
- incompatible (or not sure if compatible and no enough time/resource to investigate it) with Hadoop 3.x
so that we can publish our releases more frequently with limited resources.
To drop such components, I marked the ones that matches any of the above conditions as "to be removed" in the below list.
(Tez, Phoenix and Livy seem to be inactive recently, but they significantly improve performance or usability of Hadoop/Hive, HBase and Spark respectively, so I'd like to keep them as an exception, from my biased perspective)
So please update the BOM if your required components are going to be removed!
Components
Components in v1.5 in v3.0 alluxio 1.8.2 => 2.4.1 ambari 2.6.1 => 2.7.5 bigtop-ambari-mpack 2.6.1 => 2.7.5 bigtop_groovy 2.5.4 => 2.5.4 bigtop_jsvc 1.0.15 => 1.0.15 bigtop_tomcat 8.5.57 => removed bigtop_utils 1.5.0 => 3.0.0 elasticsearch 5.6.14 => 5.6.14 (or greater, as far as its license is ASLv2-compliant) flink 1.6.4 => 1.11.3 [3] flume 1.9.0 => removed giraph 1.2.0 => removed gpdb 5.10.0 => 5.28.5 hadoop 2.10.1 => 3.2.1 (or 3.3.x (BIGTOP-3403)? see also the discussions on BIGTOP-3280 and [4]) hbase 1.5.0 => 2.2.6 (or 2.3.3 [5], if possible) hive 2.3.6 => 3.1.2 [6] ignite_hadoop 2.7.6 => removed kafka 2.4.0 => 2.4.1 kibana 5.4.1 => 5.4.1 (or greater, as far as its license is ASLv2-compliant) livy 0.7.0 => 0.7.1 logstash 5.4.1 => 5.4.1 (or greater, as far as its license is ASLv2-compliant) mahout 0.13.0 => removed oozie 4.3.0 => 5.2.1 phoenix 4.15.0-HBase-1.5 => 5.1.0 [7] qfs 2.0.0 => removed solr 6.6.6 => 8.7.0 (or maybe 8.8.0?) spark 2.4.5 => 3.0.1 sqoop 1.4.6 => 1.4.7 sqoop2 1.99.4 => removed tez 0.9.2 => 0.10.0 ycsb 0.12.0 => 0.17.0 zeppelin 0.8.2 => 0.9.0 [8] zookeeper 3.4.13 => 3.4.14 (or 3.5+ [9] if possible)
Distros
- CentOS 7, 8 - Debian 9 (except for ppc64le, because its EOL LTS is not provided for ppc64le and its official Docker image also seems to have been removed from DockerHub as of May 2021), 10 - Fedora 33 - Ubuntu 18.04, 20.04
Some points of discussion:
- Should we still support CentOS 8 which is supposed to be EOL'd within 2021 [10]?
- Should we update Fedora to 33, the latest version as of Jan 2021?
- Should we also support Ubuntu 20.04 additionally?
Archs
- x86_64
- aarch64
- ppc64le (if the CI server is recovered)
JDK
JDK8
Also, I tentatively assigned 3.0 as its version for the following reasons, but let's discuss it too if someone has another option.
- We're going to adopt Hadoop 3.x (and hopefully Spark 3.x), so Bigtop 3.0 is aligned with them [11].
- Bigtop 2.x is (unofficially) reserved for the "Cloud Native Bigtop" (BIGTOP-3225) release line [12].
- In my opinion, this major version leap is good to impress users backward incompatibility described above.
[1]: https://lists.apache.org/thread.html/fe54be35856c8f6a23fb7d35b979e3dc90ad58baf94d15a206cdaf63%40%3Cdev.bigtop.apache.org%3E
[2]: https://lists.apache.org/thread.html/r71869111448bf5f8bc82e7031ff25d5455ab957e8e536d000026fe16%40%3Cdev.bigtop.apache.org%3E
[3]: The latest release which is officially supported by Zeppelin 0.9.0. https://twitter.com/ApacheZeppelin/status/1288855217895698433
[4]: https://github.com/apache/bigtop/pull/704
[5]: The latest release which is officially compatible with Hadoop 3.2.x as of now. https://hbase.apache.org/book.html#hadoop
[6]: The latest release which is officially compatible with Hadoop 3.x.y as of now. https://hive.apache.org/downloads.html#26-august-2019-release-312-available
[7]: The latest release which is officially compatible with HBase 2.x as of now. https://phoenix.apache.org/download.html
[8]: The latest release which officially supports Spark 3.0 as of now. https://twitter.com/ApacheZeppelin/status/1288855217895698433
[9]: HBase 2.3.3 is built with Zookeeper 3.5.7 by default and doesn't seem to be built with Zookeeper 3.4.x as it is.
[10]: https://blog.centos.org/2020/12/future-is-centos-stream/
[11]: https://issues.apache.org/jira/browse/BIGTOP-3287?focusedCommentId=17018665&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17018665
[12]: https://lists.apache.org/thread.html/e0af97d83c33bf370e25fca61aee95d0ed63193893c5f0ff7d187ebb%40%3Cdev.bigtop.apache.org%3E
