[SPARK-28294] Support `spark.history.fs.cleaner.maxNum` configuration

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 08/Jul/19 06:57
• Updated: 10/Jul/19 14:24
• Resolved: 10/Jul/19 14:20

Description

Up to now, Apache Spark maintains the event log directory by time policy, `spark.history.fs.cleaner.maxAge`. However, there are two issues.

1. Some file system has a limitation on the maximum number of files in a single directory. For example, HDFS `dfs.namenode.fs-limits.max-directory-items` is 1024 * 1024 by default.

• https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

2. Spark is sometimes unable to to clean up some old log files due to permission issues.

To handle both (1) and (2), this issue aims to support an additional number policy configuration for the event log directory, `spark.history.fs.cleaner.maxNum`. Spark can try to keep the number of files in the event log directory according to this policy.

Issue Links

• links to: GitHub Pull Request #25072