[TEZ-2724] Tez Client keeps on showing old status when application is finished but RM is shutdown

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.5.4
• Target Version/s: 0.5.5
• Assignee: Jeff Zhang
• Reporter: Jeff Zhang
• Created: 17/Aug/15 05:22
• Updated: 01/Nov/16 23:53

Description

From the logs, it seems the ipc retry interval is set as 20 seconds and ipc max retries is 45. This means that the client will retry the RPC connection for total 900 (20*45) seconds. And in this period, the application may already complete and RM Restarting may be triggered as said in the jira description. And I think the RM recovery is not enabled, so even the new RM is restarted, the original application info is lost, that means the client can never get the correct application report which makes it showing the old status forever.

15/05/07 19:13:43 INFO ipc.Client: Retrying connect to server: maint22-tez12/100.79.80.19:52822. Already tried 26 time(s); maxRetries=45
Deleted /user/hadoopqa/Input1

RUNNING: call D:\hdp\hadoop-2.6.0.2.2.6.0-2782\bin\hdfs.cmd dfs -ls /user/hadoopqa/Input2

RUNNING: call D:\hdp\hadoop-2.6.0.2.2.6.0-2782\bin\hdfs.cmd dfs  -rm -r -skipTrash /user/hadoopqa/Input2

15/05/07 19:14:03 INFO ipc.Client: Retrying connect to server: maint22-tez12/100.79.80.19:52822. Already tried 27 time(s); maxRetries=45

Configuration to reproduce this issue

• disable generic application history (yarn.timeline-service.generic-application-history.enabled)
• disable rm recovery (yarn.resourcemanager.recovery.enabled)
• increase the ipc retry interval and max retry (ipc.client.connect.retry.interval & ipc.client.connect.max.retries)