[HADOOP-17324] Don't relocate org.bouncycastle in shaded client jars

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: build
• Labels: pull-request-available
• Target Version/s: 3.3.1, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 22/Oct/20 21:33
• Updated: 11/Feb/24 13:14
• Resolved: 11/Nov/20 16:34

Description

When downstream apps depend on hadoop-client-api, hadoop-client-runtime and hadoop-client-minicluster, it seems the MiniYARNCluster could have issue because org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException is not in any of the above jars.

Error:  Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException
Error:  	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
Error:  	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
Error:  	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
Error:  	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:862)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:353)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:127)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:488)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:321)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.beforeAll(BaseYarnClusterSuite.scala:94)
Error:  	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
Error:  	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
Error:  	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
Error:  	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:61)
Error:  	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
Error:  	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
Error:  	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
Error:  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Error:  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Error:  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Error:  	at java.lang.Thread.run(Thread.java:748)

Issue Links

• blocks: SPARK-29250 Upgrade to Hadoop 3.3.1
• duplicates: HADOOP-17327 NPE when starting MiniYARNCluster from hadoop-client-minicluster
• is broken by: HADOOP-15832 Upgrade BouncyCastle to 1.60
• is related to: HBASE-28295 Few tests are failing due to NCDFE: org/bouncycastle/operator/OperatorCreationException
• relates to: HADOOP-17373 hadoop-client-integration-tests doesn't work when building with skipShade
• links to: GitHub Pull Request #2411