Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-23945

Cannot start the cluster using S3 as the file system

    XMLWordPrintableJSON

    Details

      Description

      high-availability.storageDir: s3:///flink/recovery
      When I performed the above configuration, the following error was reported

      Could not start cluster entrypoint KubernetesSessionClusterEntrypoint.
      org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint KubernetesSessionClusterEntrypoint.
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:201) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585) [flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint.main(KubernetesSessionClusterEntrypoint.java:61) [flink-dist_2.12-1.12.3.jar:1.12.3]
      Caused by: java.io.IOException: Could not create FileSystem for highly available storage path (s3:/flink/recovery/flink-native-k8s-session-1)
      at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:92) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:76) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:115) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292]
      at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_292]
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
      at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      ... 2 more
      Caused by: java.io.IOException: null uri host.
      at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:162) ~[?:?]
      at org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:62) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:507) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:89) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:76) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:115) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292]
      at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_292]
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
      at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      ... 2 more
      Caused by: java.lang.NullPointerException: null uri host.
      at java.util.Objects.requireNonNull(Objects.java:228) ~[?:1.8.0_292]
      at org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:72) ~[?:?]
      at org.apache.hadoop.fs.s3a.S3AFileSystem.setUri(S3AFileSystem.java:467) ~[?:?]
      at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:234) ~[?:?]
      at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:123) ~[?:?]
      at org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:62) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:507) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:89) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:76) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
      at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:115) ~[flink-dist

       

       

        Attachments

        1. 截屏2021-08-27 上午12.24.30.png
          587 kB
          钟洋洋
        2. 截屏2021-08-27 上午12.31.10.png
          599 kB
          钟洋洋
        3. 截屏2021-08-27 下午2.50.46.png
          57 kB
          钟洋洋
        4. 截屏2021-08-27 下午2.51.24.png
          882 kB
          钟洋洋

          Issue Links

            Activity

              People

              • Assignee:
                trohrmann Till Rohrmann
                Reporter:
                zhongyangyang 钟洋洋
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: