[YARN-2137] Add support for logaggregation to a path on non-default filecontext

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.0
• Component/s: log-aggregation
• Target Version/s: 2.6.0
• Assignee: Sumit Kumar
• Reporter: Sumit Kumar
• Created: 09/Jun/14 19:38
• Updated: 07/May/15 16:53
• Resolved: 07/May/15 16:53

Description

Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing.

This patch does two things:

1. moves logaggregation implementation to use only FileContext apis
2. adds support for doing log aggregation on non-default filesystem as well.
3. changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior

Issue Links

• is duplicated by: YARN-3269 Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path