Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2443 Handling logs of long-running services on YARN
  3. YARN-2583

Modify the LogDeletionService to support Log aggregation for LRS

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, AggregatedLogDeletionService will delete old logs from HDFS. It will check the cut-off-time, if all logs for this application is older than this cut-off-time. The app-log-dir from HDFS will be deleted. This will not work for LRS. We expect a LRS application can keep running for a long time.
      Two different scenarios:
      1) If we configured the rollingIntervalSeconds, the new log file will be always uploaded to HDFS. The number of log files for this application will become larger and larger. And there is no log files will be deleted.
      2) If we did not configure the rollingIntervalSeconds, the log file can only be uploaded to HDFS after the application is finished. It is very possible that the logs are uploaded after the cut-off-time. It will cause problem because at that time the app-log-dir for this application in HDFS has been deleted.

        Attachments

        1. YARN-2583.1.patch
          13 kB
          Xuan Gong
        2. YARN-2583.2.patch
          34 kB
          Xuan Gong
        3. YARN-2583.3.1.patch
          38 kB
          Xuan Gong
        4. YARN-2583.3.patch
          38 kB
          Xuan Gong
        5. YARN-2583.4.patch
          42 kB
          Xuan Gong
        6. YARN-2583.5.patch
          41 kB
          Xuan Gong
        7. YARN-2583.6.patch
          40 kB
          Zhijie Shen

          Issue Links

            Activity

              People

              • Assignee:
                xgong Xuan Gong
                Reporter:
                xgong Xuan Gong
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: