[FLINK-23945] Cannot start the cluster using S3 as the file system

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.12.3
• Fix Version/s: 1.14.0, 1.13.3, 1.12.8
• Component/s: Connectors / FileSystem
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: 钟洋洋
• Created: 24/Aug/21 11:55
• Updated: 15/Dec/21 01:40
• Resolved: 27/Aug/21 16:50

Description

high-availability.storageDir: s3:///flink/recovery
When I performed the above configuration, the following error was reported

Could not start cluster entrypoint KubernetesSessionClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint KubernetesSessionClusterEntrypoint.
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:201) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585) [flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint.main(KubernetesSessionClusterEntrypoint.java:61) [flink-dist_2.12-1.12.3.jar:1.12.3]
Caused by: java.io.IOException: Could not create FileSystem for highly available storage path (s3:/flink/recovery/flink-native-k8s-session-1)
at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:92) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:76) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:115) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_292]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
... 2 more
Caused by: java.io.IOException: null uri host.
at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:162) ~[?:?]
at org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:62) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:507) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:89) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:76) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:115) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_292]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
... 2 more
Caused by: java.lang.NullPointerException: null uri host.
at java.util.Objects.requireNonNull(Objects.java:228) ~[?:1.8.0_292]
at org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:72) ~[?:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.setUri(S3AFileSystem.java:467) ~[?:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:234) ~[?:?]
at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:123) ~[?:?]
at org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:62) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:507) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:89) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:76) ~[flink-dist_2.12-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:115) ~[flink-dist

 

 

Issue Links

• links to: GitHub Pull Request #17016