[HUDI-4281] Using hudi to build a large number of tables in spark on hive causes OOM

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.12.2
• Component/s: spark-sql
• Labels: pull-request-available
• Environment: spark on hive long running service ; spark version: 3.2.1 ; hive version: 2.3.x ; hudi version: hudi-spark3.2-bundle_2.12-0.11.0
• Story Points: 0.5
• Assignee: zhangrenhua
• Reporter: zhangrenhua

Reviewers

Ethan Guo (this is the old account; please use "yihua")

• Created: 19/Jun/22 12:13
• Updated: 25/Oct/22 14:10
• Resolved: 25/Oct/22 14:10

Description

On the long-running server of spark on hive, if a large number of tables are created, OOM will occur because the IsolatedClientLoader object cannot be released.

The Hive Client should adopt the singleton mode, which can avoid the oom caused by the unreleased IsolatedClientLoader object.

Issue Links

• links to: GitHub Pull Request #5903