[FLINK-31818] parsing error of 'security.kerberos.access.hadoopFileSystems' in flink-conf.yaml

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.17.0
• Fix Version/s: 1.18.0, 1.17.1
• Component/s: Runtime / Configuration
• Labels: bug pull-request-available
• Assignee: Lee SeungMin
• Reporter: Lee SeungMin
• Created: 17/Apr/23 05:57
• Updated: 17/Apr/23 15:17
• Resolved: 17/Apr/23 15:17

Description

There is a parsing error when I gave two or more hdfs namenodes URI separated by commas as the value of key attribute 'security.kerberos.access.hadoopFileSystems'.

 

For example, I set this key attribute and value like below in flink-conf.yaml,

security.kerberos.access.hadoopFileSystems: hdfs://hadoop-nn1.testurl.com:8020,hdfs://hadoop-nn2.testurl.com:8020 

 

then, the slash "/" is missing in second URI in parsed value

hdfs://hadoop-nn1.testurl.com:8020,hdfs:/hadoop-nn2.testurl.com:8020

 

 

Received error message is here.

Caused by: org.apache.flink.util.FlinkRuntimeException: java.io.IOException: Incomplete HDFS URI, no host: hdfs://hadoop-nn1.testurl.com:8020,hdfs:/hadoop-nn2.testurl.com:8020
   at org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider.lambda$getFileSystemsToAccess$2(HadoopFSDelegationTokenProvider.java:168) ~[flink-dist-1.17.0.jar:1.17.0]
   at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_362]
   at org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider.getFileSystemsToAccess(HadoopFSDelegationTokenProvider.java:157) ~[flink-dist-1.17.0.jar:1.17.0]
   at org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider.lambda$obtainDelegationTokens$1(HadoopFSDelegationTokenProvider.java:113) ~[flink-dist-1.17.0.jar:1.17.0]
   at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_362]
   at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_362]
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1966) ~[hadoop-common-2.10.0-khp-20210414.jar:?]
   at org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.java:108) ~[flink-dist-1.17.0.jar:1.17.0]
   at org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.lambda$obtainDelegationTokensAndGetNextRenewal$1(DefaultDelegationTokenManager.java:228) ~[flink-dist-1.17.0.jar:1.17.0]
   ... 13 more
Caused by: java.io.IOException: Incomplete HDFS URI, no host: hdfs://hadoop-nn1.testurl.com:8020,hdfs:/hadoop-nn2.testurl.com:8020
   at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:156) ~[hadoop-hdfs-client-2.10.0-khp-20210414.jar:?]
   at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3241) ~[hadoop-common-2.10.0-khp-20210414.jar:?]
   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:122) ~[hadoop-common-2.10.0-khp-20210414.jar:?]
   at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3290) ~[hadoop-common-2.10.0-khp-20210414.jar:?]
   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3258) ~[hadoop-common-2.10.0-khp-20210414.jar:?]
   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:471) ~[hadoop-common-2.10.0-khp-20210414.jar:?]
   at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356) ~[hadoop-common-2.10.0-khp-20210414.jar:?]
   at org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider.lambda$getFileSystemsToAccess$2(HadoopFSDelegationTokenProvider.java:163) ~[flink-dist-1.17.0.jar:1.17.0]
   at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_362]
   at org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider.getFileSystemsToAccess(HadoopFSDelegationTokenProvider.java:157) ~[flink-dist-1.17.0.jar:1.17.0]
   at org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider.lambda$obtainDelegationTokens$1(HadoopFSDelegationTokenProvider.java:113) ~[flink-dist-1.17.0.jar:1.17.0]
   at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_362]
   at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_362]
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1966) ~[hadoop-common-2.10.0-khp-20210414.jar:?]
   at org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.java:108) ~[flink-dist-1.17.0.jar:1.17.0]
   at org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.lambda$obtainDelegationTokensAndGetNextRenewal$1(DefaultDelegationTokenManager.java:228) ~[flink-dist-1.17.0.jar:1.17.0]
   ... 13 more 

 

Issue Links

• links to: GitHub Pull Request #22409