Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14125

s3guard tool tests aren't isolated; can't run in parallel

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HADOOP-13345
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3, test
    • Labels:


      The S3GuardToolTestBase don't parallelize and break other tests. This can surface if you do a full run with -Ds3guard and -Ddynamo.

      1. many of the test paths they create are being requested with absolute paths, e.g "/test-diff".
      2. the base class doesn't set up a per-forked-JUnit-test runner path in the bucket
      3. and there's no cleanup at the end of each test case; teardown() is empty.

      Ideally, the tests should be made child classes of AbstractS3ATestBase, with its post-run cleanup, If that can't be done, then the tests must

      1. use S3ATestUtils.createTestPath(super.getTestPath()) to create the base test path
      2. clean up that dir in teardown if the FS instance is non-null. ContractTestUtils.cleanup() can do this.

      If it happens that the tests cannot run in parallel with others, then the build must be changed to exclude them from the parallel test run phase & include them in the serialized section.




            • Assignee:
              mackrorysd Sean Mackrory
              stevel@apache.org Steve Loughran


              • Created:

                Issue deployment