Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0
-
Reviewed
Description
Description of Problem
Applications which call YARN RMAdminCLI (i.e. YARN ResourceManager client) synchronously can be blocked for up to 15 minutes with the default configuration of "yarn.resourcemanager.connect.max-wait.ms"; this is not an issue in of itself, but there is a non-retryable IllegalArgumentException exception thrown within the YARN ResourceManager client that is getting swallowed & treated as a retryable "connection exception" meaning that it gets retried for 15 minutes.
The purpose of this JIRA (and PR) is to modify the YARN client so that it does not retry on this non-retryable exception.
Background Information
YARN ResourceManager client treats connection exceptions as retryable & with the default value of "yarn.resourcemanager.connect.max-wait.ms" will attempt to connect to the ResourceManager for up to 15 minutes when facing "connection exceptions". This arguably makes sense because connection exceptions are in some cases transient & can be recovered from without any action needed from the client. See example below where YARN ResourceManager client was able to recover from connection issues that resulted from the ResourceManager process being down.
> yarn rmadmin -refreshNodes
22/06/28 14:40:17 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033
22/06/28 14:40:18 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:40:19 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:40:20 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...
22/06/28 14:40:27 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:40:28 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:40:29 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...
22/06/28 14:40:37 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:40:37 INFO retry.RetryInvocationHandler: java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 1 failover attempts. Trying to failover after sleeping for 41061ms.
22/06/28 14:41:19 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:41:20 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...
22/06/28 14:41:28 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:41:28 INFO retry.RetryInvocationHandler: java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 2 failover attempts. Trying to failover after sleeping for 25962ms.>> Success is silent in client logs, but can be seen in the ResourceManager logs <<
Then there are cases where the YARN ResourceManager client will stop retrying because it has encountered a non retryable exception. Some examples:
- client is configured with SIMPLE auth when ResourceManager is configured with KERBEROS auth
- this RemoteException is not a transient failure & will not recover without the client taking action to modify their configuration, this is why it fails immediately
- the exception is coming from ResourceManager server-side & will occur once the client successfully calls the ResourceManager
> yarn rmadmin -refreshNodes
22/07/12 15:20:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033
refreshNodes: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[KERBEROS]
- client & server as configured with KERBEROS auth but the client has not kinit
- this SaslException is not a transient failure & will not recover without the client taking action to modify their configuration, this is why it fails immediately
- the exception is coming from client-side & will occur before the client even attempts to call the ResourceManager
> yarn rmadmin -refreshNodes
22/07/12 15:20:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033
22/07/12 15:20:33 WARN ipc.Client: Exception encountered while connecting to the server
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:408)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:629)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:423)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:825)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:820)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:820)
at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:423)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1617)
at org.apache.hadoop.ipc.Client.call(Client.java:1448)
at org.apache.hadoop.ipc.Client.call(Client.java:1401)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy7.refreshNodes(Unknown Source)
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes(ResourceManagerAdministrationProtocolPBClientImpl.java:145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy8.refreshNodes(Unknown Source)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.refreshNodes(RMAdminCLI.java:349)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.refreshNodes(RMAdminCLI.java:423)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.handleRefreshNodes(RMAdminCLI.java:917)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.run(RMAdminCLI.java:816)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.main(RMAdminCLI.java:1027)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:162)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 34 more
refreshNodes: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033;
The Problem
When the client has:
- kerberos enabled by setting "hadoop.security.authentication = kerberos" in "core-site.xml"
- a bad kerberos configuration where "yarn.resourcemanager.principal" is unset or malformed in "yarn-site.xml"
This bad configuration can never successfully connect to the ResourceManager & therefore should result in a non-retryable failure.
When the YARN ResourceManager client has this bad configuration an IllegalArugmentException gets thrown (in org.apache.hadoop.security.SaslRpcClient) but then swallowed by an IOException (in org.apache.hadoop.ipc.Client) that gets treated as a retryable failure & therefore will be retried for 15 minutes:
> yarn rmadmin -refreshNodes
22/06/28 14:23:45 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033
22/06/28 14:23:46 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:23:47 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:23:48 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...
22/06/28 14:23:54 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:23:55 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:23:56 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:23:57 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:23:58 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...
22/06/28 14:24:04 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:24:05 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8033. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/06/28 14:24:05 INFO retry.RetryInvocationHandler: java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 1 failover attempts. Trying to failover after sleeping for 27166ms.
22/06/28 14:24:32 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033; , while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 2 failover attempts. Trying to failover after sleeping for 22291ms.
22/06/28 14:24:54 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033; , while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 3 failover attempts. Trying to failover after sleeping for 24773ms.
22/06/28 14:25:19 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033; , while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 4 failover attempts. Trying to failover after sleeping for 39187ms.
...
22/06/28 14:36:50 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033; , while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 26 failover attempts. Trying to failover after sleeping for 26235ms.
22/06/28 14:37:16 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033; , while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 27 failover attempts. Trying to failover after sleeping for 40535ms.
22/06/28 14:37:57 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033; , while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 28 failover attempts. Trying to failover after sleeping for 26721ms.
22/06/28 14:38:23 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033; , while invoking ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes over null after 29 failover attempts. Trying to failover after sleeping for 27641ms.
refreshNodes: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033;
This non-retryable failure should not be treated as a retryable "connection failure"
The Solution
Surface the IllegalArgumentException to the RetryInvocationHandler & have YARN RMProxy treat IllegalArugmentException as non-retryable
Note that surfacing IllegalArgumentException has the side-effect of causing the command usage to be printed here
...
refreshNodes: Failed to specify server's Kerberos principal name
Usage: yarn rmadmin [-refreshNodes [-g|graceful [timeout in seconds] -client|server]]Generic options supported are:
-conf <configuration file> specify an application configuration file
-D <property=value> define a value for a given property
-fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt <local|resourcemanager:port> specify a ResourceManager
-files <file1,...> specify a comma-separated list of files to be copied to the map reduce cluster
-libjars <jar1,...> specify a comma-separated list of jar files to be included in the classpath
-archives <archive1,...> specify a comma-separated list of archives to be unarchived on the compute machinesThe general command line syntax is:
command [genericOptions] [commandOptions]
To resolve this issue the IllegalArgumentException is swallowed & surfaced as a KerberosAuthException, this chosen because it already gets [treated as non-retryable in FailoverOnNetworkExceptionRetry]()
Note that in terms of RetryPolicy:
- non-HA YARN ResourceManager client should use OtherThanRemoteExceptionDependentRetry (but because of a bug uses FailoverOnNetworkExceptionRetry)
- HA YARN ResourceManager client uses FailoverOnNetworkExceptionRetry
The result of this change is a much quicker failure when the YARN client is misconfigured:
- non-HA YARN ResourceManager client
> yarn rmadmin -refreshNodes
22/07/13 17:36:03 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033
22/07/13 17:36:03 WARN ipc.Client: Exception encountered while connecting to the server
javax.security.sasl.SaslException: Bad Kerberos server principal configuration [Caused by java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name]
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:237)
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:159)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:397)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:630)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:424)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:825)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:821)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:821)
at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:424)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1612)
at org.apache.hadoop.ipc.Client.call(Client.java:1442)
at org.apache.hadoop.ipc.Client.call(Client.java:1395)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy7.refreshNodes(Unknown Source)
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes(ResourceManagerAdministrationProtocolPBClientImpl.java:145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy8.refreshNodes(Unknown Source)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.refreshNodes(RMAdminCLI.java:349)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.refreshNodes(RMAdminCLI.java:423)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.handleRefreshNodes(RMAdminCLI.java:917)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.run(RMAdminCLI.java:816)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.main(RMAdminCLI.java:1027)
Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:332)
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:233)
... 35 more
refreshNodes: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: Bad Kerberos server principal configuration [Caused by java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name]; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033;
- HA YARN ResourceManager client
> yarn rmadmin -refreshNodes
22/07/13 17:37:50 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033
22/07/13 17:37:50 WARN ipc.Client: Exception encountered while connecting to the server
javax.security.sasl.SaslException: Bad Kerberos server principal configuration [Caused by java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name]
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:237)
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:159)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:397)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:630)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:424)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:825) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:821)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:821)
at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:424)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1612)
at org.apache.hadoop.ipc.Client.call(Client.java:1442)
at org.apache.hadoop.ipc.Client.call(Client.java:1395)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy7.refreshNodes(Unknown Source)
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceManagerAdministrationProtocolPBClientImpl.refreshNodes(ResourceManagerAdministrationProtocolPBClientImpl.java:145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy8.refreshNodes(Unknown Source)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.refreshNodes(RMAdminCLI.java:349)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.refreshNodes(RMAdminCLI.java:423)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.handleRefreshNodes(RMAdminCLI.java:917)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.run(RMAdminCLI.java:816)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.yarn.client.cli.RMAdminCLI.main(RMAdminCLI.java:1027)
Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:332)
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:233)
... 35 more
refreshNodes: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: Bad Kerberos server principal configuration [Caused by java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name]; Host Details : local host is: "0.0.0.0/0.0.0.0"; destination host is: "0.0.0.0":8033;
Other Notes
The YARN RMProxy will return separate RetryPolicies for HA & non-HA, but the YARN client will always use the HA policy because a configuration related to Federation Failover is enabled by default. This is presumably a bug because YARN Federation is not enabled for the cluster I am testing on.
The fix is to modify HAUtil.isFederationFailoverEnabled to check if "yarn.federation.enabled" (default false) in addition to checking if "yarn.federation.failover.enabled" (default true).
Attachments
Issue Links
- links to
