[HIVE-16904] during repl load for large number of partitions the metadata file can be huge and can lead to out of memory

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: HiveServer2
• Assignee: Anishek Agarwal Assign to me
• Reporter: Anishek Agarwal
• Created: 15/Jun/17 05:37
• Updated: 21/Oct/22 07:20

Description

the metadata pertaining to a table + its partitions is stored in a single file, During repl load all the data is loaded in memory in one shot and then individual partitions processed. This can lead to huge memory overhead as the entire file is read in memory. try to deserialize the partition objects with some sort of streaming json deserializer.