Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16027

[DOC] Effective use of FS instances during S3A integration tests

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • fs/s3
    • None

    Description

      While fixing HADOOP-15819 we found that a closed fs got into the static fs cache during testing, which caused other tests to fail when the tests were running sequentially.
      We should document some best practices in the testing section on the s3 docs with the following:

      Tests using FileSystems are fastest if they can recycle the existing FS instance from the same JVM. If you do that, you MUST NOT close or do unique configuration on them. If you want a guarantee of 100% isolation or an instance with unique config, create a new instance
      which you MUST close in the teardown to avoid leakage of resources.
      Do not add FileSystem instances (with e.g org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that will be modified or closed during the test runs. This can cause other tests to fail when using the same modified or closed FS instance. For more details see HADOOP-15819.

      Attachments

        1. HADOOP-16027.001.patch
          2 kB
          Gabor Bota
        2. HADOOP-16027.002.patch
          2 kB
          Gabor Bota

        Activity

          People

            gabor.bota Gabor Bota
            gabor.bota Gabor Bota
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: