[YARN-4720] Skip unnecessary NN operations in log aggregation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Jun Gong
• Reporter: Ming Ma
• Created: 23/Feb/16 01:29
• Updated: 06/Jan/17 08:03
• Resolved: 26/Feb/16 16:58

Description

Log aggregation service could have unnecessary NN operations in the following scenarios:

• No new local log has been created since the last upload for the long running service scenario.
• NM uses ContainerLogAggregationPolicy that skips log aggregation for certain containers.

In the following code snippet, even though pendingContainerInThisCycle is empty, it still creates the writer and then removes the file later. Thus it introduces unnecessary create/getfileinfo/delete NN calls when NM doesn't aggregate logs for an app.

AppLogAggregatorImpl.java
......
        writer =
            new LogWriter(this.conf, this.remoteNodeTmpLogFileForApp,
                this.userUgi);
......
      for (ContainerId container : pendingContainerInThisCycle) {
......
      }
......
            if (remoteFS.exists(remoteNodeTmpLogFileForApp)) {
              if (rename) {
                remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath);
              } else {
                remoteFS.delete(remoteNodeTmpLogFileForApp, false);
              }
            }
......