[FLINK-12623] Flink on yarn encountered AMRMClientImpl does not update AMRM token properly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.9.0
• Component/s: Connectors / Hadoop Compatibility
• Assignee: dalongliu
• Reporter: dalongliu
• Due: 25/May/19
• Created: 25/May/19 06:35
• Updated: 16/Jul/19 10:48
• Resolved: 16/Jul/19 10:47

Description

Hi, all! When my task running on yarn dependency flink verison is 1.7.1, I encountered the following problem:

[org.apache.hadoop.ipc.Client ] >>> msg=Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1547520251214_0582_000001sg [2019-05-25 03:31:04.163] [WARN] [AMRM Heartbeater thread] [org.apache.hadoop.ipc.Client ] >>> msg=Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1547520251214_0582_000001sg [2019-05-25 03:31:04.168] [INFO] [AMRM Heartbeater thread] [org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider] >>> msg=Failing over to rm1sg [2019-05-25 03:31:04.168] [INFO] [AMRM Heartbeater thread] [org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider] >>> msg=Failing over to rm1sg [2019-05-25 03:31:30.143] [INFO] [AMRM Heartbeater thread] [org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider] >>> msg=Failing over to rm2sg [2019-05-25 03:31:30.143] [INFO] [AMRM Heartbeater thread] [org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider] >>> msg=Failing over to rm2sg

Refering to the yarn issueAMRMClientImpl does not update AMRM token properly , I know this problem is 2.4.1 version yarn bug, this bug is fixed in 2.7.0 verison yarn. However, flink shaded hadoop version is 2.4.1, so I think this bug, flink should shade 2.7.0 version hadoop or a higher version, is my idea correct?

 

Issue Links

• is caused by: YARN-3103 AMRMClientImpl does not update AMRM token properly