[TEZ-4296] Consider using listStatusIterator instead of listStatus in DatePartitionedLogger

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.1
• Assignee: Harish JP
• Reporter: Rajesh Balamohan
• Created: 21/Feb/21 22:55
• Updated: 19/May/21 01:33
• Resolved: 19/May/21 01:33

Description

DatePartitionedLogger should make use of listStatusIterator instead of listStatus to avoid OOM.

https://github.com/apache/tez/blob/master/tez-plugins/tez-protobuf-history-plugin/src/main/java/org/apache/tez/dag/history/logging/proto/DatePartitionedLogger.java#L163

e.g /warehouse/tablespace/managed/hive/sys.db/query_data/date=x-y-z had way too may files and listing them with DatePartitionedLogger OOMed.

Issue Links

• is related to: HADOOP-17709 ChecksumFileSystem does not filter .crc files in listStatusIterator
• links to: GitHub Pull Request #124