[HADOOP-17098] Reduce Guava dependency in Hadoop source code

• Type: Task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Assignee: Ahmed Hussein
• Reporter: Ahmed Hussein
• Created: 29/Jun/20 15:59
• Updated: 09/Dec/21 16:05

Description

Relying on Guava implementation in Hadoop has been painful due to compatibility and vulnerability issues.
Guava updates tend to break/deprecate APIs. This made It hard to maintain backward compatibility within hadoop versions and clients/downstreams.

With 3.x uses java8+, the java 8 features should preferred to Guava, reducing the footprint, and giving stability to source code.

This jira should serve as an umbrella toward an incremental effort to reduce the usage of Guava in the source code and to create subtasks to replace Guava classes with Java features.

Furthermore, it will be good to add a rule in the pre-commit build to warn against introducing a new Guava usage in certain modules.

Any one willing to take part in this code refactoring has to:

1. Focus on one module at a time in order to reduce the conflicts and the size of the patch. This will significantly help the reviewers.
2. Run all the unit tests related to the module being affected by the change. It is critical to verify that any change will not break the unit tests, or cause a stable test case to become flaky.
3. Merge should be done to the following branches:  trunk, branch-3.3, branch-3.2, branch-3.1

 

A list of sub tasks replacing Guava APIs with java8 features:

com.google.common.io.BaseEncoding#base64()	java.util.Base64
com.google.common.io.BaseEncoding#base64Url()	java.util.Base64
com.google.common.base.Joiner.on()	                        java.lang.String#join() or 
                                                                                         java.util.stream.Collectors#joining()
com.google.common.base.Optional#of()	                java.util.Optional#of()
com.google.common.base.Optional#absent()	        java.util.Optional#empty()
com.google.common.base.Optional#fromNullable()	java.util.Optional#ofNullable()
com.google.common.base.Optional	                        java.util.Optional
com.google.common.base.Predicate	                        java.util.function.Predicate
com.google.common.base.Function	                        java.util.function.Function
com.google.common.base.Supplier	                        java.util.function.Supplier

 

I also vote for the replacement of Precondition with either a wrapper, or Apache commons lang.

I believe you guys have dealt with Guava compatibilities in the past and probably have better insights. Any thoughts? weichiu, gabor.bota, [e-mail], ayushtkn, busbey, jeagles, kihwal

 

Issue Links

• fixes: DRILL-7854 When writing to S3 "WriteOperationHelper.newUploadPartRequest" throws "NoSuchMethodError"
• incorporates: HADOOP-17743 Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, hadoop-tools and cloud-storage projects
• incorporates: HDFS-16054 Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project
• incorporates: MAPREDUCE-7350 Replace Guava Lists usage by Hadoop's own Lists in hadoop-mapreduce-project
• incorporates: YARN-10805 Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project
• incorporates: HADOOP-17753 Keep restrict-imports-enforcer-rule for Guava Lists in hadoop-main pom
• relates to: HADOOP-17967 Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom

Sub-Tasks

• Replace Guava Predicate with Java8+ Predicate: Resolved
• Replace Guava Supplier with Java8+ Supplier in Hadoop: Resolved
• Replace Guava Supplier with Java8+ Supplier in MAPREDUCE: Resolved
• Replace Guava Supplier with Java8+ Supplier in hdfs: Resolved
• Replace Guava Function with Java8+ Function: Resolved
• Replace Guava Optional with Java8+ Optional: Resolved
• Update the checkstyle config to ban some guava functions: Resolved
• add guava BaseEncoding to illegalClasses: Resolved
• remove guava Preconditions from Hadoop-common-project modules: Resolved
• implement non-guava Precondition checkNotNull: Resolved
• Replace Guava Preconditions to avoid Guava dependency: In Progress
• Add unguava implementation for Joiner in hadoop.StringUtils: Patch Available
• Implement wrapper for guava newArrayList and newLinkedList: Resolved
• Replace Guava initialization of Lists.newArrayList: Resolved
• Create Classes to wrap Guava code replacement: Open
• Replace Guava Sets usage by Hadoop's own Sets in hadoop-common and hadoop-tools: Resolved
• Replace Guava.Splitter with common.util.StringUtils: Open
• Add checkstyle rule to prevent further usage of Guava classes: Resolved
• Replace Guava Sets usage by Hadoop's own Sets in hadoop-hdfs-project: Resolved
• Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project: Resolved
• Replace Guava Sets usage by Hadoop's own Sets in hadoop-mapreduce-project: Resolved
• implement non-guava Precondition checkArgument: Resolved
• implement non-guava Precondition checkState: Resolved
• Provide alternative to Guava VisibleForTesting: Resolved
• Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-common-project modules: Resolved
• Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules: Resolved
• Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-cloud-storage-project and hadoop-mapreduce-project modules: Resolved
• hadoop-auth module cannot import non-guava implementation in hatoop util: Resolved
• Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-tools modules: Resolved
• Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules: Resolved
• unguava: remove Preconditions from hdfs-projects module: Resolved
• unguava: remove Preconditions from hadoop-yarn-project modules: Resolved
• unguava: remove Preconditions from hadoop-tools modules: Resolved
• Add restrict-imports-enforcer-rule for Guava Preconditions in hadoop-main pom: Resolved