[HADOOP-14195] CredentialProviderFactory$getProviders is not thread-safe

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
• Component/s: security
• Target Version/s: 2.8.1
• Assignee: Vihang Karajgaonkar
• Reporter: Vihang Karajgaonkar
• Created: 17/Mar/17 18:05
• Updated: 08/Jun/17 00:09
• Resolved: 23/Mar/17 21:02

Description

Multi-threaded access to CredentialProviderFactory is not thread-safe because java.util.ServiceLoader is not thread-safe (as noted in its Java doc). Thanks to jzhuge I was able to reproduce this issue but creating a simple multi-threaded application which executes the following code in parallel.

for (int i = 0; i < ITEMS; i++) {
	    futures.add(executor.submit(new Callable<Void>() {
		@Override
		public Void call() throws Exception {
		    boolean found = false;
		    for (CredentialProviderFactory factory : serviceLoader) {
			CredentialProvider kp = factory.createProvider(uri, conf);
			if (kp != null) {
			    result.add(kp);
			    found = true;
			    break;
			}
		    }
		    if (!found) {
			throw new IOException(Thread.currentThread() + "No CredentialProviderFactory for " + uri);
		    } else {
			System.out.println(Thread.currentThread().getName() + " found credentialProvider for " + path);
		    }
		    return null;
		}
	    }));
	}

I see the following exception trace when I execute the above code.

java.util.concurrent.ExecutionException: java.util.NoSuchElementException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at TestCredentialProvider.main(TestCredentialProvider.java:58)
Caused by: java.util.NoSuchElementException
	at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:615)
	at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:590)
	at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:61)
	at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
	at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
	at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
	at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
	at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I also see a NPE sometimes

java.util.concurrent.ExecutionException: java.lang.NullPointerException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at TestCredentialProvider.main(TestCredentialProvider.java:58)
Caused by: java.lang.NullPointerException
	at java.util.ServiceLoader.parse(ServiceLoader.java:304)
	at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
	at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
	at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
	at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
	at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• is related to: SLIDER-888 intermittent errors when accessing key store password during localization of cert stores