[PHOENIX-6698] hive-connector will take long time to generate splits for large phoenix tables.

• Type: Improvement
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 5.1.0
• Fix Version/s: connectors-6.0.0
• Component/s: hive-connector
• Assignee: jichen
• Reporter: jichen
• Created: 27/Apr/22 09:21
• Updated: 15/Jul/22 06:27

Description

In our production environment, hive-phoenix connector  will take nearly 30-40 minutes to generate splits for large phoenix table, which has more than 2048 regions.it is because in class PhoenixInputFormat, function  'generateSplits' only uses one thread to generate splits for each scan. My proposal is to use multi-thread to generate splits in parallel. the proposal has been validated in our production environment.by  changing code to generate splits  in parallel with 24 threads, the time cost is reduced to 2 minutes. 

Issue Links

• links to: GitHub Pull Request #79