[GOBBLIN-2032] Support making log dir sink optional

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-cluster
• Assignee: Hung Tran
• Reporter: William Lo
• Created: 01/Apr/24 21:38
• Updated: 08/Apr/24 20:10

Description

In Gobblin cluster, there is a configuration to set the log sink in order to create a log copier service which copies logs to HDFS from Yarn cluster.

Since Yarn has a built in log aggregator, we should have a way to disable this log sink since it becomes redundant, so we should look into ways to make the log directory optional.