[BIGTOP-3813] Incompatible guava & commons-lang3 Versions in Oozie Sharelib

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.0.2, 3.2.0, 3.1.2
• Component/s: hadoop
• Flags: Patch
• Assignee: William Horn
• Reporter: William Horn
• Created: 13/Sep/22 15:07
• Updated: 28/Sep/22 11:43
• Resolved: 28/Sep/22 10:14

Description

Within the Oozie do-component-build script, the version of dependent Hadoop components (Hadoop, Spark, Hive, etc) are updated to the target Bigtop versions. These target versions are included in the Oozie sharelib. However, the build does not update the version of dependencies for these components, namely Spark in the case we ran into. We found that when Spark was running in yarn-cluster mode, and the oozie sharelib was uploaded to HDFS, any Oozie workflows with Spark tasks would fail due to guava and commons-lang3 versions included in the oozie sharelib that were still targetting the old component versions.

Updating these dependencies in the oozie sharelib resolved the workflow failures:

guava: 14.0.1 -> 27.0-jre

commons-lang3: 3.3.9 -> 3.9

Issue Links

• is related to: BIGTOP-3833 Fix failure of smoke-tests of Oozie