[SPARK-42511] Spark MasterWebUI and WorkerWebUI fail to start when NSSDB used as keystore, getting java.security.KeyStoreException: PKCS11 not found.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0, 3.3.1, 3.3.2
• Component/s: Spark Core, Spark Submit
• Assignee: Unassigned
• Reporter: SHOBHIT SHUKLA
• Created: 21/Feb/23 13:13
• Updated: 21/Feb/23 13:24

Description

When we are running Spark by setting up below SSL configurations, Spark masterwebui and workerwebui is fail to start.

"spark.ssl.enabled":"true",
"spark.ssl.keyStore":"/opt/ibm/jdk/conf/security/nss.fips.cfg",
"spark.ssl.keyStorePassword":"<keystore passwd>",
"spark.ssl.keyStoreType":"PKCS11"

Errors :

23/02/21 12:29:43 INFO Master: Running Spark version 3.3.1
23/02/21 12:29:43 ERROR MasterWebUI: Failed to bind MasterWebUI
java.security.KeyStoreException: PKCS11 not found
at java.base/java.security.KeyStore.getInstance(KeyStore.java:878)
at org.sparkproject.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:46)
at org.sparkproject.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
at org.sparkproject.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
at org.sparkproject.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.sparkproject.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.sparkproject.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.sparkproject.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.sparkproject.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.sparkproject.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.sparkproject.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
at org.sparkproject.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.sparkproject.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.spark.ui.JettyUtils$.newConnector$1(JettyUtils.scala:303)
at org.apache.spark.ui.JettyUtils$.sslConnect$1(JettyUtils.scala:322)
at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$4(JettyUtils.scala:326)
at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$4$adapted(JettyUtils.scala:326)
at org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2401)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2393)
at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$2(JettyUtils.scala:326)
at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$2$adapted(JettyUtils.scala:315)
at scala.Option.map(Option.scala:230)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:315)
at org.apache.spark.ui.WebUI.initServer(WebUI.scala:144)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:153)
at org.apache.spark.deploy.master.Master.onStart(Master.scala:138)
at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:120)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:839)
Caused by: java.security.NoSuchAlgorithmException: PKCS11 KeyStore not available
at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
at java.base/java.security.Security.getImpl(Security.java:719)
at java.base/java.security.KeyStore.getInstance(KeyStore.java:875)
... 37 more

content of nss fips config file.

name = NSS-FIPS
nssLibraryDirectory = /usr/lib64
nssSecmodDirectory = /etc/pki/nssdb
nssDbMode = readOnly
nssModule = fips

attributes(*,CKO_SECRET_KEY,CKK_GENERIC_SECRET)=

Unknown macro: { CKA_SIGN=true }