[HADOOP-18508] support multiple s3a integration test runs on same bucket in parallel

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.9
• Fix Version/s: 3.5.0, 3.4.1
• Component/s: fs/s3, test
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 26/Oct/22 13:25
• Updated: 13/Sep/24 16:38
• Resolved: 14/Jun/24 18:49

Description

It is now possible to provide a job ID in the maven "job.id" property
hadoop-aws test runs to isolate paths under a the test bucket
under which all tests will be executed.

This will allow independent builds in different source trees
to test against the same bucket in parallel, and is designed for
CI testing.

Example:

mvn verify -Dparallel-tests -Droot.tests.enabled=false -Djob.id=1
mvn verify -Droot.tests.enabled=false -Djob.id=2

• Root tests must be be disabled to stop them cleaning up
  the test paths of other test runs.
• Do still regularly run the root tests just to force cleanup
  of the output of any interrupted test suites.

Issue Links

• causes: HADOOP-19273 S3A: Commiter ITests failing unless job.id set
• links to: GitHub Pull Request #5081