Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15953

AliyunOSS: make AliyunCredentialsProvider compatible with the required constructor

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.10.0, 3.2.0, 2.9.2, 3.0.3, 3.3.0, 3.1.2
    • Fix Version/s: None
    • Component/s: fs/oss
    • Labels:
      None

      Description

      Hadoop aliyun module uses AliyunCredentialsProvider as default CredentialsProvider if user do not set fs.oss.credentials.provider. However, if user set fs.oss.credentials.provider to org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider, exception will be thrown like below:

      java.io.IOException: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider constructor exception. A class specified in fs.oss.credentials.provider must provide an accessible constructor accepting URI and Configuration, or an accessible default constructor.
      
       at org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.getCredentialsProvider(AliyunOSSUtils.java:131)
       at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:154)
       at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:344)
       at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3302)
       at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
       at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3351)
       at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
       at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
       at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
       at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
       at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
       at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
       at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
       at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
       at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
       at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
       at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider.<init>()
       at java.lang.Class.getConstructor0(Class.java:3082)
       at java.lang.Class.getDeclaredConstructor(Class.java:2178)
       at org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.getCredentialsProvider(AliyunOSSUtils.java:125)
       ... 23 more
      

      Because AliyunCredentialsProvider does not have corresponding constructor.

      public class AliyunCredentialsProvider implements CredentialsProvider {
        private Credentials credentials = null;
      
        public AliyunCredentialsProvider(Configuration conf) throws IOException {
      
      --------------------------------------------------
      try {
        credentials =
            (CredentialsProvider)credClass.getDeclaredConstructor(
                URI.class, Configuration.class).newInstance(uri, conf);
      } catch (NoSuchMethodException | SecurityException e) {
        credentials =
            (CredentialsProvider)credClass.getDeclaredConstructor()
            .newInstance();
      }
      

       

      Although the documentation says you should provide a `specified class must provide an accessible constructor accepting java.net.URI and org.apache.hadoop.conf.Configuration` if you set this configuration, we should make 

      AliyunCredentialsProvider compatible with this.

      <property>
        <name>fs.oss.credentials.provider</name>
        <description>
          Class name of a credentials provider that implements
          com.aliyun.oss.common.auth.CredentialsProvider. Omit if using access/secret keys
          or another authentication mechanism. The specified class must provide an
          accessible constructor accepting java.net.URI and
          org.apache.hadoop.conf.Configuration, or an accessible default constructor.
        </description>
      </property>
      

        Attachments

        1. HADOOP-15953.002.patch
          2 kB
          wujinhu
        2. HADOOP-15953.001.patch
          2 kB
          wujinhu

          Activity

            People

            • Assignee:
              wujinhu wujinhu
              Reporter:
              wujinhu wujinhu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: