[KAFKA-13487] Create a topic partition directory based on the size of the directory

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: log
• Assignee: Unassigned
• Reporter: Gongwenzhou
• Created: 29/Nov/21 08:15
• Updated: 06/Dec/21 16:39

Description

In the current multi-directory scenario of `log.dirs`, logManager determines which base directory the new partition directory should be in according to the number of subdirectories in each base directory. However, in the actual production environment, the file size generated by each subject partition is different. As a result, the disk usage of the base directory is unbalanced.
We can provide a configuration to determine the strategy of logManager to create a new partition directory, as follows:
Broker-level configuration items:log.directory.select.strategy, there are two values: partition and size
partition: Sort by the number of directories under each base directory (log.dirs) (the current version of the new partition creation strategy)
size: Sort by the size of each directory (log.dirs), the smallest directory has the highest allocation right

Issue Links

• links to: GitHub Pull Request #11548