[HDDS-7991] Intermittent failure in FS contract tests

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: test
• Labels: pull-request-available
• Target Version/s: 1.4.0, 1.3.1
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 18/Feb/23 17:54
• Updated: 14/Mar/23 21:32
• Resolved: 22/Feb/23 09:54

Description

Intermittent test failures due to unexpectedly found files.

https://github.com/adoroszlai/ozone-build-results/blob/master/2023/01/03/19325/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRootDir.txt

java.lang.AssertionError: Deleted file: unexpectedly found /testmkdirdepth1 as  FileStatus{path=o3fs://bucket55106.volume05976/testmkdirdepth1; isDirectory=true; modification_time=1672746966478; access_time=1672746966478; owner=runner; group=runner; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:1004)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:733)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:706)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:688)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertDeleted(AbstractFSContractTestBase.java:372)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testMkDirDepth1(AbstractContractRootDirectoryTest.java:74)

https://github.com/adoroszlai/ozone-build-results/blob/master/2023/02/02/19901/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRootDir.txt

java.lang.AssertionError: Deleted file: unexpectedly found o3fs://bucket07060.volume35574/test as  FileStatus{path=o3fs://bucket07060.volume35574/test; isDirectory=true; modification_time=1675366936086; access_time=1675366936086; owner=runner; group=runner; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:1004)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:733)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:196)

https://github.com/adoroszlai/ozone-build-results/blob/master/2023/02/16/20197/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDelete.txt

java.lang.AssertionError: Deleted file: unexpectedly found o3fs://bucket29507.volume05478/test/testDeleteEmptyDirNonRecursive as  FileStatus{path=o3fs://bucket29507.volume05478/test/testDeleteEmptyDirNonRecursive; isDirectory=true; modification_time=1676579169907; access_time=1676579169907; owner=runner; group=runner; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:1004)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:733)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:706)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:688)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertDeleted(AbstractFSContractTestBase.java:372)
	at org.apache.hadoop.fs.contract.AbstractContractDeleteTest.testDeleteEmptyDirNonRecursive(AbstractContractDeleteTest.java:37)

https://github.com/adoroszlai/ozone-build-results/blob/master/2023/02/17/20204/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDelete.txt

ITestOzoneContractDelete.testDeleteNonEmptyDirRecursive  Time elapsed: 0.184 s  <<< FAILURE!
java.lang.AssertionError: Deleted file: unexpectedly found o3fs://bucket17885.volume94145/test/testDeleteNonEmptyDirRecursive as  FileStatus{path=o3fs://bucket17885.volume94145/test/testDeleteNonEmptyDirRecursive; isDirectory=true; modification_time=1676597571285; access_time=1676597571285; owner=runner; group=runner; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:1004)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:733)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:706)
	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:688)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertDeleted(AbstractFSContractTestBase.java:372)
	at org.apache.hadoop.fs.contract.AbstractContractDeleteTest.testDeleteNonEmptyDirRecursive(AbstractContractDeleteTest.java:93)

Issue Links

• fixes: HDDS-7940 Intermittent failure in testFileDelete (O3FS)
• is caused by: HDDS-7253 Ozone fs report error when key names containing '/'
• is related to: HDDS-7871 Fix false positive in KeyManagerImpl#createFakeDirIfShould()
• links to: GitHub Pull Request #4287