[SLIDER-888] intermittent errors when accessing key store password during localization of cert stores

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: appmaster, security
• Assignee: Jonathan Maron
• Reporter: Jonathan Maron
• Created: 21/May/15 20:06
• Updated: 17/Mar/17 18:54
• Resolved: 01/Jul/15 15:57

Description

Under certain conditions exceptions are generated during localization of certificate stores to containers:

2015-05-18 20:35:38,533 [RoleLaunchService-010] ERROR appmaster.RoleLaunchService - Exception thrown while trying to start ACCUMULO_MONITOR: java.util.NoSuchElementException
java.util.NoSuchElementException
at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:59)
at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:349)
at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:439)
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57)
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1946)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1926)
at org.apache.slider.server.services.security.AbstractSecurityStoreGenerator.getStorePassword(AbstractSecurityStoreGenerator.java:64)
at org.apache.slider.server.services.security.KeystoreGenerator.generate(KeystoreGenerator.java:43)
at org.apache.slider.server.services.security.StoresGenerator.generateSecurityStores(StoresGenerator.java:48)
at org.apache.slider.providers.agent.AgentProviderService.generateSecurityStores(AgentProviderService.java:561)
at org.apache.slider.providers.agent.AgentProviderService.localizeContainerSecurityStores(AgentProviderService.java:545)
at org.apache.slider.providers.agent.AgentProviderService.buildContainerLaunchContext(AgentProviderService.java:453)
at org.apache.slider.server.appmaster.RoleLaunchService$RoleLauncher.run(RoleLaunchService.java:201)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

and

2015-05-18 20:28:42,393 [RoleLaunchService-008] ERROR appmaster.RoleLaunchService - Exception thrown while trying to start ACCUMULO_TSERVER: java.io.IOException: Configuration problem with provider path.
java.io.IOException: Configuration problem with provider path.
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1966)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1926)
at org.apache.slider.server.services.security.AbstractSecurityStoreGenerator.getStorePassword(AbstractSecurityStoreGenerator.java:64)
at org.apache.slider.server.services.security.KeystoreGenerator.generate(KeystoreGenerator.java:43)
at org.apache.slider.server.services.security.StoresGenerator.generateSecurityStores(StoresGenerator.java:48)
at org.apache.slider.providers.agent.AgentProviderService.generateSecurityStores(AgentProviderService.java:561)
at org.apache.slider.providers.agent.AgentProviderService.localizeContainerSecurityStores(AgentProviderService.java:545)
at org.apache.slider.providers.agent.AgentProviderService.buildContainerLaunchContext(AgentProviderService.java:453)
at org.apache.slider.server.appmaster.RoleLaunchService$RoleLauncher.run(RoleLaunchService.java:201)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No CredentialProviderFactory for jceks://hdfs/user/hrt_qa/accumulo-test_read_write_ssl.jceks in hadoop.security.credential.provider.path
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:66)
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1946)
... 11 more

Issue Links

• relates to: HADOOP-14195 CredentialProviderFactory$getProviders is not thread-safe