[YARN-4755] Optimize sending appACLsUpdated event to TimelineServer while recovering completed applications

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Naganarasimha G R
• Reporter: Rohith Sharma K S
• Created: 02/Mar/16 10:02
• Updated: 03/Mar/16 07:02

Description

In method RMAppManager#createAndPopulateNewRMApp, appACLsUpdated is sent to timelineserver for every application that get created.

 private RMAppImpl createAndPopulateNewRMApp(
      ApplicationSubmissionContext submissionContext, long submitTime,
      String user, boolean isRecovery) throws YarnException {
		//
                //
    String appViewACLs = submissionContext.getAMContainerSpec()
        .getApplicationACLs().get(ApplicationAccessType.VIEW_APP);
    rmContext.getSystemMetricsPublisher().appACLsUpdated(
        application, appViewACLs, System.currentTimeMillis());
    return application;
  }

Say if we have 10K completed applications to recover, 30K events will be generated i.e app_created, app_finished and app_acl_updated. For completed applications, I think need not to send app-acl-updated event with which gradually reduce load on the dispatcher.

Eventhough MultiDispatcher is used to publish timeline events, it is bottle neck when max-completed is configured very high value may be 100K.