Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14125

s3guard tool tests aren't isolated; can't run in parallel

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HADOOP-13345
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3, test
    • Labels:
      None

      Description

      The S3GuardToolTestBase don't parallelize and break other tests. This can surface if you do a full run with -Ds3guard and -Ddynamo.

      1. many of the test paths they create are being requested with absolute paths, e.g "/test-diff".
      2. the base class doesn't set up a per-forked-JUnit-test runner path in the bucket
      3. and there's no cleanup at the end of each test case; teardown() is empty.

      Ideally, the tests should be made child classes of AbstractS3ATestBase, with its post-run cleanup, If that can't be done, then the tests must

      1. use S3ATestUtils.createTestPath(super.getTestPath()) to create the base test path
      2. clean up that dir in teardown if the FS instance is non-null. ContractTestUtils.cleanup() can do this.

      If it happens that the tests cannot run in parallel with others, then the build must be changed to exclude them from the parallel test run phase & include them in the serialized section.

        Attachments

          Activity

            People

            • Assignee:
              mackrorysd Sean Mackrory
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: