Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14195

CredentialProviderFactory$getProviders is not thread-safe

    Details

    • Target Version/s:

      Description

      Multi-threaded access to CredentialProviderFactory is not thread-safe because java.util.ServiceLoader is not thread-safe (as noted in its Java doc). Thanks to John Zhuge I was able to reproduce this issue but creating a simple multi-threaded application which executes the following code in parallel.

      for (int i = 0; i < ITEMS; i++) {
      	    futures.add(executor.submit(new Callable<Void>() {
      		@Override
      		public Void call() throws Exception {
      		    boolean found = false;
      		    for (CredentialProviderFactory factory : serviceLoader) {
      			CredentialProvider kp = factory.createProvider(uri, conf);
      			if (kp != null) {
      			    result.add(kp);
      			    found = true;
      			    break;
      			}
      		    }
      		    if (!found) {
      			throw new IOException(Thread.currentThread() + "No CredentialProviderFactory for " + uri);
      		    } else {
      			System.out.println(Thread.currentThread().getName() + " found credentialProvider for " + path);
      		    }
      		    return null;
      		}
      	    }));
      	}
      

      I see the following exception trace when I execute the above code.

      java.util.concurrent.ExecutionException: java.util.NoSuchElementException
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at TestCredentialProvider.main(TestCredentialProvider.java:58)
      Caused by: java.util.NoSuchElementException
      	at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:615)
      	at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:590)
      	at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:61)
      	at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
      	at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
      	at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
      	at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
      	at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      I also see a NPE sometimes

      java.util.concurrent.ExecutionException: java.lang.NullPointerException
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at TestCredentialProvider.main(TestCredentialProvider.java:58)
      Caused by: java.lang.NullPointerException
      	at java.util.ServiceLoader.parse(ServiceLoader.java:304)
      	at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
      	at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
      	at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
      	at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
      	at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
      	at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      1. TestCredentialProvider.java
        2 kB
        Vihang Karajgaonkar
      2. HADOOP-14195.003.patch
        2 kB
        Vihang Karajgaonkar
      3. HADOOP-14195.002.patch
        2 kB
        Vihang Karajgaonkar
      4. HADOOP-14195.001.patch
        2 kB
        Vihang Karajgaonkar

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
          Hide
          jzhuge John Zhuge added a comment -

          Since s3a is in 2.7, this should be backported. I will do it shortly.

          Show
          jzhuge John Zhuge added a comment - Since s3a is in 2.7, this should be backported. I will do it shortly.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Seems serious. Should this go back to branch 2.7?

          Show
          stevel@apache.org Steve Loughran added a comment - Seems serious. Should this go back to branch 2.7?
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11451 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11451/)
          HADOOP-14195. CredentialProviderFactory$getProviders is not thread-safe. (jzhuge: rev 128015584d69492806fd1700c8f840d78aa9c729)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProviderFactory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11451 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11451/ ) HADOOP-14195 . CredentialProviderFactory$getProviders is not thread-safe. (jzhuge: rev 128015584d69492806fd1700c8f840d78aa9c729) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProviderFactory.java
          Hide
          jzhuge John Zhuge added a comment -

          Committed to trunk, branch-2, and branch-2.8.

          Thanks Vihang Karajgaonkar for the contribution!

          Show
          jzhuge John Zhuge added a comment - Committed to trunk, branch-2, and branch-2.8. Thanks Vihang Karajgaonkar for the contribution!
          Hide
          jzhuge John Zhuge added a comment -

          +1 LGTM

          I will commit in a few days if no objection.

          Show
          jzhuge John Zhuge added a comment - +1 LGTM I will commit in a few days if no objection.
          Hide
          jzhuge John Zhuge added a comment -

          Sure, I will review the patch in the next few days. Thanks for the hard work, it is tough to reliably reproduce race conditions.

          Show
          jzhuge John Zhuge added a comment - Sure, I will review the patch in the next few days. Thanks for the hard work, it is tough to reliably reproduce race conditions.
          Hide
          vihangk1 Vihang Karajgaonkar added a comment -

          Thanks John Zhuge for letting me know. Can you please review the patch? I could not add a test-case which reproduces the race-condition consistently. If you have any ideas please let me know and I would be happy to add it as a test-case. I have tried the following approaches but I am not getting the expected results consistently.

          1. Introduce a testcase in TestCredentialProviderFactory which spawns a threadpool which calls CredentialProviderFactory.getProviders(conf) from each thread. --> This test case works consistently when run individually but produces false positives when run along with other tests (I think junit schedules the threads along with other testcases such that race condition is not reproduced)
          2. Call CredentialProviderFactory.getProviders(conf) in a loop from each thread 100 times to improve the chances of race-condition. But it didn't help either.

          Thanks!

          Show
          vihangk1 Vihang Karajgaonkar added a comment - Thanks John Zhuge for letting me know. Can you please review the patch? I could not add a test-case which reproduces the race-condition consistently. If you have any ideas please let me know and I would be happy to add it as a test-case. I have tried the following approaches but I am not getting the expected results consistently. 1. Introduce a testcase in TestCredentialProviderFactory which spawns a threadpool which calls CredentialProviderFactory.getProviders(conf) from each thread. --> This test case works consistently when run individually but produces false positives when run along with other tests (I think junit schedules the threads along with other testcases such that race condition is not reproduced) 2. Call CredentialProviderFactory.getProviders(conf) in a loop from each thread 100 times to improve the chances of race-condition. But it didn't help either. Thanks!
          Hide
          jzhuge John Zhuge added a comment -

          Both unit test failures are known. TestSFTPFileSystem#testFileExists failed often lately. It is tracked by HADOOP-14206.

          Show
          jzhuge John Zhuge added a comment - Both unit test failures are known. TestSFTPFileSystem#testFileExists failed often lately. It is tracked by HADOOP-14206 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 13s trunk passed
          +1 compile 21m 33s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 51s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 16m 58s the patch passed
          +1 javac 16m 58s the patch passed
          +1 checkstyle 0m 34s the patch passed
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 37s the patch passed
          +1 javadoc 0m 49s the patch passed
          -1 unit 8m 15s hadoop-common in the patch failed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          73m 30s



          Reason Tests
          Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem
            hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14195
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859759/HADOOP-14195.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 29ad58d90a44 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2841666
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11866/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11866/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11866/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 13s trunk passed +1 compile 21m 33s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 51s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 16m 58s the patch passed +1 javac 16m 58s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 37s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 8m 15s hadoop-common in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 73m 30s Reason Tests Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem   hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859759/HADOOP-14195.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 29ad58d90a44 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2841666 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11866/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11866/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11866/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vihangk1 Vihang Karajgaonkar added a comment -

          Based on what I see in the console logs in https://builds.apache.org/job/PreCommit-HADOOP-Build/11863/console I see that pre-commit is trying to apply the test java application instead of the patch. Re-attaching the patch again.

          Show
          vihangk1 Vihang Karajgaonkar added a comment - Based on what I see in the console logs in https://builds.apache.org/job/PreCommit-HADOOP-Build/11863/console I see that pre-commit is trying to apply the test java application instead of the patch. Re-attaching the patch again.
          Hide
          vihangk1 Vihang Karajgaonkar added a comment -

          Attaching the Java application which reproduces the race condition. The exception is not thrown when the patch is applied.

          Show
          vihangk1 Vihang Karajgaonkar added a comment - Attaching the Java application which reproduces the race condition. The exception is not thrown when the patch is applied.
          Hide
          vihangk1 Vihang Karajgaonkar added a comment -

          Attaching the second version for the patch which fixes the white-spaces. This patch fixes a race-condition in java.util.ServiceLoader when it is iterated in parallel by multiple threads. Its very hard to write a testcase to consistently reproduce the race-condition. I tried adding a test case which creates a Threadpool and calls CredentialProviderFactory.getProviders(conf)) from multiple threads. It works when run individually but creates false positives when run along with other threads. This could be due to the way Junit schedules the threads for each testcase.

          I tested the patch manually with the attached test java application which reproduces the race-condition consistently.

          The JUnit test TestKDiag is working for me locally. John Zhuge Do you know how can I reproduce this test failure?

          Show
          vihangk1 Vihang Karajgaonkar added a comment - Attaching the second version for the patch which fixes the white-spaces. This patch fixes a race-condition in java.util.ServiceLoader when it is iterated in parallel by multiple threads. Its very hard to write a testcase to consistently reproduce the race-condition. I tried adding a test case which creates a Threadpool and calls CredentialProviderFactory.getProviders(conf)) from multiple threads. It works when run individually but creates false positives when run along with other threads. This could be due to the way Junit schedules the threads for each testcase. I tested the patch manually with the attached test java application which reproduces the race-condition consistently. The JUnit test TestKDiag is working for me locally. John Zhuge Do you know how can I reproduce this test failure?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 37s trunk passed
          +1 compile 10m 30s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 1m 31s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 10m 21s the patch passed
          +1 javac 10m 21s the patch passed
          -0 checkstyle 0m 39s hadoop-common-project/hadoop-common: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          -1 whitespace 0m 0s The patch 10 line(s) with tabs.
          +1 findbugs 1m 37s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 8m 14s hadoop-common in the patch failed.
          +1 asflicense 0m 44s The patch does not generate ASF License warnings.
          56m 0s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14195
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859633/HADOOP-14195.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 929c614cbb97 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6c399a8
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 37s trunk passed +1 compile 10m 30s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 1m 31s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 10m 21s the patch passed +1 javac 10m 21s the patch passed -0 checkstyle 0m 39s hadoop-common-project/hadoop-common: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 24s the patch passed -1 whitespace 0m 0s The patch 10 line(s) with tabs. +1 findbugs 1m 37s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 8m 14s hadoop-common in the patch failed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 56m 0s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859633/HADOOP-14195.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 929c614cbb97 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6c399a8 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11860/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vihangk1 Vihang Karajgaonkar added a comment -

          Attaching the patch which synchronizes access to serviceLoader in CredentialProviderFactory

          Show
          vihangk1 Vihang Karajgaonkar added a comment - Attaching the patch which synchronizes access to serviceLoader in CredentialProviderFactory
          Hide
          vihangk1 Vihang Karajgaonkar added a comment -

          Attaching the correct version Test code to reproduce this error

          Show
          vihangk1 Vihang Karajgaonkar added a comment - Attaching the correct version Test code to reproduce this error
          Hide
          jzhuge John Zhuge added a comment -

          More and more applications start to access cloud filesystem directly using URI s3a:// or adl://. Configuration.getPassword is called during construction of each fs instance to get credentials, thus more likely in multiple threads. Configuration.getPassword uses service loader to load credential providers.

          Here is a real-world Hive exception backtrace:

          Caused by: java.util.NoSuchElementException
          	at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:59)
          	at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
          	at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
          	at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
          	at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57)
          	at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950)
          	at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSAccessKeys(S3AFileSystem.java:374)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:175)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2696)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2733)
          	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2715)
          	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382)
          	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
          	at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385)
          	at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:372)
          	at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:252)
          	at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
          	at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81)
          	at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
          	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:67)
          

          SLIDER-888 reported a similar issue.

          So somewhere along the stack, we need to serialize the call.

          Show
          jzhuge John Zhuge added a comment - More and more applications start to access cloud filesystem directly using URI s3a:// or adl:// . Configuration.getPassword is called during construction of each fs instance to get credentials, thus more likely in multiple threads. Configuration.getPassword uses service loader to load credential providers. Here is a real-world Hive exception backtrace: Caused by: java.util.NoSuchElementException at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:59) at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357) at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393) at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57) at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950) at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930) at org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSAccessKeys(S3AFileSystem.java:374) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:175) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2696) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2733) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2715) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385) at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:372) at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:252) at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95) at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81) at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:67) SLIDER-888 reported a similar issue. So somewhere along the stack, we need to serialize the call.
          Hide
          vihangk1 Vihang Karajgaonkar added a comment -

          Adding the simple multi-threaded Java application to reproduce this issue. You will need to add common-lang, common-collections, commons-logging, guava and hadoop-common jars in the classpath to be able to run this application.

          Show
          vihangk1 Vihang Karajgaonkar added a comment - Adding the simple multi-threaded Java application to reproduce this issue. You will need to add common-lang, common-collections, commons-logging, guava and hadoop-common jars in the classpath to be able to run this application.

            People

            • Assignee:
              vihangk1 Vihang Karajgaonkar
              Reporter:
              vihangk1 Vihang Karajgaonkar
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development