Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-15796

ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs caching is disabled

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 3.1.1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Sometimes we get java.io.IOException: s3a://TEST-BUCKET: FileSystem is closed! when running tests, because of filesystem caching: an fs instance is closed from a test teardown while another test is still using it. (That should be fixed in another issue.)

      To avoid test errors introduced by closed fs I've disabled fs caching while testing. This caused the following issue in lots of test:

      [ERROR] testPartitionsResolution(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing)  Time elapsed: 1.994 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.S3AFileSystem cannot be cast to org.apache.hadoop.fs.s3a.MockS3AFileSystem
      	at org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase.lookupWrapperFS(StagingTestBase.java:145)
      	at org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$JobCommitterTest.setupJob(StagingTestBase.java:294)
      

      We should fix the casting during these tests.

      To switch of fs caching during a test (mvn verify) run, add the following to the config:

        <property>
          <name>fs.s3a.impl.disable.cache</name>
          <value>true</value>
        </property>
      

        Attachments

          Activity

            People

            • Assignee:
              gabor.bota Gabor Bota
              Reporter:
              gabor.bota Gabor Bota
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: