Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14125

s3guard tool tests aren't isolated; can't run in parallel

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • HADOOP-13345
    • HADOOP-13345
    • fs/s3, test
    • None

    Description

      The S3GuardToolTestBase don't parallelize and break other tests. This can surface if you do a full run with -Ds3guard and -Ddynamo.

      1. many of the test paths they create are being requested with absolute paths, e.g "/test-diff".
      2. the base class doesn't set up a per-forked-JUnit-test runner path in the bucket
      3. and there's no cleanup at the end of each test case; teardown() is empty.

      Ideally, the tests should be made child classes of AbstractS3ATestBase, with its post-run cleanup, If that can't be done, then the tests must

      1. use S3ATestUtils.createTestPath(super.getTestPath()) to create the base test path
      2. clean up that dir in teardown if the FS instance is non-null. ContractTestUtils.cleanup() can do this.

      If it happens that the tests cannot run in parallel with others, then the build must be changed to exclude them from the parallel test run phase & include them in the serialized section.

      Attachments

        Activity

          People

            mackrorysd Sean Mackrory
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: