[HBASE-26349] Improve recent change to IntegrationTestLoadCommonCrawl

• Type: Test
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.5.0, 3.0.0-alpha-2
• Component/s: integration tests, test
• Hadoop Flags: Reviewed
• Assignee: Andrew Kyle Purtell
• Reporter: Andrew Kyle Purtell
• Created: 11/Oct/21 20:16
• Updated: 20/Oct/21 18:33
• Resolved: 19/Oct/21 21:20

Description

Using BufferedMutator wasn't a good idea because we assign client timestamps, and the store loop is fast enough that on rare occasion two temporally adjacent URLs in the set of WARCs are equivalent and the timestamp does not advance, leading later to a rare false positive CORRUPT finding.

While making changes, support direct S3N paths as input paths on the command line.

Issue Links

• fixes: HBASE-26335 Minor improvements to IntegrationTestLoadCommonCrawl
• links to: GitHub Pull Request #3744