[PHOENIX-6694] Avoid unnecessary calls of fetching table meta data to region servers holding the system tables in batch oriented jobs in spark or hive otherwise those RS become hotspot

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: hive-connector, spark-connector
• Assignee: Rajeshbabu Chintaguntla
• Reporter: Rajeshbabu Chintaguntla
• Created: 26/Apr/22 06:45
• Updated: 03/Jun/22 00:20
• Resolved: 03/Jun/22 00:20

Description

Currently we are preparing the query plan in both data source and partition readers which is creating new connection in each worker and job initialisation which unnecessarily touch basing all both system catalog table, system stats table as well as meta. When there are jobs with millions of parallel workers hotspot the region servers holding the meta and system catalog as well system stats table. So if we share the same query plan between the workers which can avoid the hotspot.

Issue Links

• links to: GitHub Pull Request #78
• links to: GitHub Pull Request #80

Sub-Tasks

• Add support of skipping the system tables existence check during connection initialisation: Resolved