[HIVE-20528] hive onprem-s3 replication is slow

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: repl
• Target Version/s: 4.0.0
• Assignee: mahesh kumar behera
• Reporter: mahesh kumar behera
• Created: 11/Sep/18 04:24
• Updated: 21/Oct/22 07:20

Description

Hive onprem - s3 replication was initiated for 1TB tpc-ds schema ( ~ 250GB orc file size ) and 16hrs into the run it still says only 10% is complete. Out of the 24 tables, only 4 tables are replicated.

The same schema in onprem-onprem use case get done in almost ~2hrs.

Sub-Tasks

• Statistics update in S3 is taking time at target side during REPL Load: Open
• Each metastore operation in S3 taking time due to pre event authorization: Open
• Adding notification is taking time in S3 replication: Open
• File operation at target side during S3 replication slowing down the replication: Open