Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-16115

Aliyun oss filesystem could not work with plugin mechanism

    XMLWordPrintableJSON

    Details

      Description

      From release-1.9, Flink suggest users to load all filesystem with plugin, including oss. However, it could not work for oss filesystem. The root cause is it does not shade the org.apache.flink.runtime.fs.hdfs and org.apache.flink.runtime.util. So they will always be loaded by system classloader and throw the following exceptions.

       

      2020-02-17 17:28:47,247 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Could not start cluster entrypoint StandaloneSessionClusterEntrypoint.
      org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.
              at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:187)
              at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
              at org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:64)
      Caused by: java.lang.NoSuchMethodError: org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.<init>(Lorg/apache/flink/fs/shaded/hadoop3/org/apache/hadoop/fs/FileSystem;)V
              at org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:85)
              at org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:61)
              at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:441)
              at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:362)
              at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
              at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:100)
              at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:89)
              at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:125)
              at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:305)
              at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:263)
              at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:207)
              at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
              at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
              at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
              ... 2 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fly_in_gis Yang Wang
                Reporter:
                fly_in_gis Yang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m