[HUDI-1310] Corruption Block Handling too slow in S3

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: performance, writer-core
• Labels: sev:critical user-support-issues
• Assignee: Volodymyr Burenin
• Reporter: Balaji Varadarajan
• Created: 01/Oct/20 11:46
• Updated: 24/Mar/21 04:58
• Resolved: 23/Mar/21 15:50

Description

The logic to figure out next valid starting block offset is too slow when run in S3. 

I have bolded the log message that takes long time to appear. 

 

 

36589 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner - Scanning log file HoodieLogFile{pathStr='s3a://robinhood-encrypted-hudi-data-cove/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045', fileLen=0}
36590 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader - Found corrupted block in file HoodieLogFile{pathStr='s3a://robinhood-encrypted-hudi-data-cove/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045', fileLen=0} with block size(3723305) running past EOF
36684 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader - Log HoodieLogFile{pathStr='s3a://xxxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045', fileLen=0} has a corrupted block at 14
44515 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader - Next available block in HoodieLogFile{pathStr='s3a://xxxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045', fileLen=0} starts at 3723319
44566 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner - Found a corrupt block in s3a://xxxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045