Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16586

ITestS3GuardFsck, others fails when run using a local metastore

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • fs/s3
    • None
    • Reviewed

    Description

      Most of these tests fail if running against a local metastore with a ClassCastException.

      Not sure if these tests are intended to work with dynamo only. The fix (either ignore in case of other metastores or fix the test) would depend on the original intent.

      -------------------------------------------------------------------------------
      Test set: org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
      -------------------------------------------------------------------------------
      Tests run: 12, Failures: 0, Errors: 11, Skipped: 1, Time elapsed: 34.653 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
      testIDetectParentTombstoned(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 3.237 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentTombstoned(ITestS3GuardFsck.java:190)
      
      testIDetectDirInS3FileInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 1.827 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectDirInS3FileInMs(ITestS3GuardFsck.java:214)
      
      testIDetectLengthMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.819 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectLengthMismatch(ITestS3GuardFsck.java:311)
      
      testIEtagMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.832 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIEtagMismatch(ITestS3GuardFsck.java:373)
      
      testIDetectFileInS3DirInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.752 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectFileInS3DirInMs(ITestS3GuardFsck.java:238)
      
      testIDetectModTimeMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 4.103 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectModTimeMismatch(ITestS3GuardFsck.java:346)
      
      testIDetectNoMetadataEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 3.017 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoMetadataEntry(ITestS3GuardFsck.java:113)
      
      testIDetectNoParentEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.821 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoParentEntry(ITestS3GuardFsck.java:136)
      
      testINoEtag(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 4.493 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testINoEtag(ITestS3GuardFsck.java:403)
      
      testIDetectParentIsAFile(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.782 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentIsAFile(ITestS3GuardFsck.java:163)
      
      testTombstonedInMsNotDeletedInS3(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 3.008 s  <<< ERROR!
      java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testTombstonedInMsNotDeletedInS3(ITestS3GuardFsck.java:432)
      

      ITestPartialRenamesDeletes

      [ERROR]   ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 » FileNotFound Path ...
      [ERROR]   ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 » FileNotFound Path ...
      [ERROR]   ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316 » FileNotFound
      [ERROR]   ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316 » FileNotFound
      
      -------------------------------------------------------------------------------
      Test set: org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
      -------------------------------------------------------------------------------
      Tests run: 24, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 19.043 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
      testRenameParentPathNotWriteable[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)  Time elapsed: 0.947 s  <<< ERROR!
      java.io.FileNotFoundException: Expected file: not found s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir in s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272
              at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
      Caused by: java.io.FileNotFoundException: Path s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir is recorded as deleted by S3Guard at 2019-09-24T07:20:53.835Z
              at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
      
      testRenameDirFailsInDelete[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)  Time elapsed: 0.825 s  <<< ERROR!
      java.io.FileNotFoundException: Path s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=false]-single-5232/writableDir is recorded as deleted by S3Guard at 2019-09-24T07:20:55.696Z
              at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
      
      testRenameParentPathNotWriteable[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)  Time elapsed: 0.938 s  <<< ERROR!
      java.io.FileNotFoundException: Expected file: not found s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir in s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609
              at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
      Caused by: java.io.FileNotFoundException: Path s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir is recorded as deleted by S3Guard at 2019-09-24T07:21:02.152Z
              at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
      
      testRenameDirFailsInDelete[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)  Time elapsed: 0.834 s  <<< ERROR!
      java.io.FileNotFoundException: Path s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-3392/writableDir is recorded as deleted by S3Guard at 2019-09-24T07:21:03.843Z
              at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
      

      I also see AccessDenied exceptions in the test logs. Have not debugged this further.

      ITestS3GuardToolLocal

      [ERROR] Errors:
      [ERROR]   ITestS3GuardToolLocal.testInitNegativeRead:176->AbstractS3GuardToolTestBase.runToFailure:150->AbstractS3GuardToolTestBase.run:137 » IllegalArgument
      
      Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.577 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
      testInitNegativeRead(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)  Time elapsed: 0.182 s  <<< ERROR!
      java.lang.IllegalArgumentException: bucket
              at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal.testInitNegativeRead(ITestS3GuardToolLocal.java:176)
      

      ITestS3GuardToolLocal ends up skipping all tests if running with -Ddynamo, or without -Ds3guard - so these tests are exercised with the -Ds3guard option only.

      Attachments

        Issue Links

          Activity

            People

              iwasakims Masatake Iwasaki
              sseth Siddharth Seth
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: