[HADOOP-16586] ITestS3GuardFsck, others fails when run using a local metastore

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: fs/s3
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Masatake Iwasaki
• Reporter: Siddharth Seth
• Created: 18/Sep/19 17:52
• Updated: 27/Jan/24 06:33
• Resolved: 19/May/20 23:48

Description

Most of these tests fail if running against a local metastore with a ClassCastException.

Not sure if these tests are intended to work with dynamo only. The fix (either ignore in case of other metastores or fix the test) would depend on the original intent.

-------------------------------------------------------------------------------
Test set: org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
-------------------------------------------------------------------------------
Tests run: 12, Failures: 0, Errors: 11, Skipped: 1, Time elapsed: 34.653 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
testIDetectParentTombstoned(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 3.237 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentTombstoned(ITestS3GuardFsck.java:190)

testIDetectDirInS3FileInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 1.827 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectDirInS3FileInMs(ITestS3GuardFsck.java:214)

testIDetectLengthMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.819 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectLengthMismatch(ITestS3GuardFsck.java:311)

testIEtagMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.832 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIEtagMismatch(ITestS3GuardFsck.java:373)

testIDetectFileInS3DirInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.752 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectFileInS3DirInMs(ITestS3GuardFsck.java:238)

testIDetectModTimeMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 4.103 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectModTimeMismatch(ITestS3GuardFsck.java:346)

testIDetectNoMetadataEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 3.017 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoMetadataEntry(ITestS3GuardFsck.java:113)

testIDetectNoParentEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.821 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoParentEntry(ITestS3GuardFsck.java:136)

testINoEtag(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 4.493 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testINoEtag(ITestS3GuardFsck.java:403)

testIDetectParentIsAFile(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 2.782 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentIsAFile(ITestS3GuardFsck.java:163)

testTombstonedInMsNotDeletedInS3(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)  Time elapsed: 3.008 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
  at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testTombstonedInMsNotDeletedInS3(ITestS3GuardFsck.java:432)

ITestPartialRenamesDeletes

[ERROR]   ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 » FileNotFound Path ...
[ERROR]   ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 » FileNotFound Path ...
[ERROR]   ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316 » FileNotFound
[ERROR]   ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316 » FileNotFound

-------------------------------------------------------------------------------
Test set: org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
-------------------------------------------------------------------------------
Tests run: 24, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 19.043 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
testRenameParentPathNotWriteable[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)  Time elapsed: 0.947 s  <<< ERROR!
java.io.FileNotFoundException: Expected file: not found s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir in s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272
        at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
Caused by: java.io.FileNotFoundException: Path s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir is recorded as deleted by S3Guard at 2019-09-24T07:20:53.835Z
        at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)

testRenameDirFailsInDelete[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)  Time elapsed: 0.825 s  <<< ERROR!
java.io.FileNotFoundException: Path s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=false]-single-5232/writableDir is recorded as deleted by S3Guard at 2019-09-24T07:20:55.696Z
        at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)

testRenameParentPathNotWriteable[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)  Time elapsed: 0.938 s  <<< ERROR!
java.io.FileNotFoundException: Expected file: not found s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir in s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609
        at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
Caused by: java.io.FileNotFoundException: Path s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir is recorded as deleted by S3Guard at 2019-09-24T07:21:02.152Z
        at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)

testRenameDirFailsInDelete[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)  Time elapsed: 0.834 s  <<< ERROR!
java.io.FileNotFoundException: Path s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-3392/writableDir is recorded as deleted by S3Guard at 2019-09-24T07:21:03.843Z
        at org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)

I also see AccessDenied exceptions in the test logs. Have not debugged this further.

ITestS3GuardToolLocal

[ERROR] Errors:
[ERROR]   ITestS3GuardToolLocal.testInitNegativeRead:176->AbstractS3GuardToolTestBase.runToFailure:150->AbstractS3GuardToolTestBase.run:137 » IllegalArgument

Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.577 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
testInitNegativeRead(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)  Time elapsed: 0.182 s  <<< ERROR!
java.lang.IllegalArgumentException: bucket
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal.testInitNegativeRead(ITestS3GuardToolLocal.java:176)

ITestS3GuardToolLocal ends up skipping all tests if running with -Ddynamo, or without -Ds3guard - so these tests are exercised with the -Ds3guard option only.

Issue Links

• links to: GitHub Pull Request #1950