[HIVE-25205] Reduce overhead of adding write notification log during batch loading of partition..

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Component/s: Hive, HiveServer2
• Labels: performance
• External issue URL: https://github.com/apache/hive/pull/2357
• Assignee: mahesh kumar behera
• Reporter: mahesh kumar behera
• Created: 07/Jun/21 03:51
• Updated: 15/Aug/23 06:33
• Resolved: 05/Aug/21 09:30

Description

During batch loading of partition the write notification logs are added for each partition added. This is causing delay in execution as the call to HMS is done for each partition. This can be optimised by adding a new API in HMS to send a batch of partition and then this batch can be added together to the backend database. Once we have a batch of notification log, at HMS side, code can be optimised to add the logs using single call to backend RDBMS. 

Issue Links

• causes: IMPALA-11925 Add new api in MetastoreServiceHandler for add_write_notification_log_in_batch method
• is related to: HIVE-24649 Optimise Hive::addWriteNotificationLog for large data inserts