[HIVE-20534] File operation at target side during S3 replication slowing down the replication

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: repl
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: mahesh kumar behera
• Reporter: mahesh kumar behera
• Created: 11/Sep/18 04:34
• Updated: 21/Oct/22 07:20

Description

1. Check is there during create partition for existence of partition location (in add partitions core method in metastore.java). It’s not required as we would have created the directory and copied the required files to it.
2. Creating qualified directory name (convertAddSpecToMetaPartition method in hive.java)– File system is access to check if the path provided is fully qualified or not. Not sure why it’s taking 1-2 seconds.