[SPARK-32097] Allow reading history log files from multiple directories

• Type: Wish
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.4.5
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Gaurangi Saxena
• Created: 25/Jun/20 02:36
• Updated: 08/Sep/20 16:21

Description

Our service dynamically creates short-lived YARN clusters in cloud. Spark applications run on these dynamically created clusters. Log data for these applications is stored on a remote file-system. We want a static instance of SparkHistoryServer to view information on jobs that ran on these clusters. We use glob because we cannot have a static list of directories where the log files reside. 

Issue Links

• links to: [Github] Pull Request #29630 (Gaurangi94)