Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4720

Skip unnecessary NN operations in log aggregation

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Log aggregation service could have unnecessary NN operations in the following scenarios:

      • No new local log has been created since the last upload for the long running service scenario.
      • NM uses ContainerLogAggregationPolicy that skips log aggregation for certain containers.

      In the following code snippet, even though pendingContainerInThisCycle is empty, it still creates the writer and then removes the file later. Thus it introduces unnecessary create/getfileinfo/delete NN calls when NM doesn't aggregate logs for an app.

      AppLogAggregatorImpl.java
      ......
              writer =
                  new LogWriter(this.conf, this.remoteNodeTmpLogFileForApp,
                      this.userUgi);
      ......
            for (ContainerId container : pendingContainerInThisCycle) {
      ......
            }
      ......
                  if (remoteFS.exists(remoteNodeTmpLogFileForApp)) {
                    if (rename) {
                      remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath);
                    } else {
                      remoteFS.delete(remoteNodeTmpLogFileForApp, false);
                    }
                  }
      ......
      

        Attachments

        1. YARN-4720.01.patch
          1 kB
          Jun Gong
        2. YARN-4720.02.patch
          6 kB
          Jun Gong
        3. YARN-4720.03.patch
          8 kB
          Jun Gong
        4. YARN-4720.04.patch
          9 kB
          Jun Gong
        5. YARN-4720.05.patch
          9 kB
          Jun Gong

          Activity

            People

            • Assignee:
              hex108 Jun Gong
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: