[IMPALA-11720] FileMetadataLoaderTest is flaky due to IOException: Filesystem closed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.2.0
• Fix Version/s: Impala 4.2.0
• Component/s: Frontend
• Epic Color: ghx-label-5
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 11/Nov/22 01:59
• Updated: 15/Nov/22 01:40
• Resolved: 15/Nov/22 01:40

Description

IMPALA-11699 recently reverts the changes in FileMetadataLoaderTest that extends it to  FrontendTestBase. This causes tests in it become flaky if run individually. E.g. the following commands all fail:

(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testRecursiveLoading)
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testHudiParquetLoading)
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testIcebergLoading)
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testAcidMinorCompactionLoading)
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory)
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testSkipHiddenDirectories)

and the failures are the same, i.e. due to java.io.IOException: Filesystem closed

$ (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory)
...
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.601 s <<< FAILURE! - in org.apache.impala.catalog.FileMetadataLoaderTest
[ERROR] testLoadMissingDirectory(org.apache.impala.catalog.FileMetadataLoaderTest)  Time elapsed: 1.565 s  <<< ERROR!
java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:483)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1662)
	at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1266)
	at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1249)
	at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
	at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
	at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2145)
	at org.apache.impala.common.FileSystemUtil.listLocatedStatusIterator(FileSystemUtil.java:804)
	at org.apache.impala.common.FileSystemUtil$RecursingIterator.<init>(FileSystemUtil.java:975)
	at org.apache.impala.common.FileSystemUtil$RecursingIterator.<init>(FileSystemUtil.java:958)
	at org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:768)
	at org.apache.impala.catalog.FileMetadataLoader.load(FileMetadataLoader.java:187)
	at org.apache.impala.catalog.FileMetadataLoaderTest.testLoadMissingDirectory(FileMetadataLoaderTest.java:192)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)

However, any test runs after the failure will succeed. If we run the test twice (by adding -Dsurefire.rerunFailingTestsCount=2), the second run also succeed:

$ (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory -Dsurefire.rerunFailingTestsCount=2)
...
[WARNING] Flakes: 
[WARNING] org.apache.impala.catalog.FileMetadataLoaderTest.testLoadMissingDirectory(org.apache.impala.catalog.FileMetadataLoaderTest)
[ERROR]   Run 1: FileMetadataLoaderTest.testLoadMissingDirectory:192 » IO Filesystem closed
[INFO]   Run 2: PASS
[INFO] 
[INFO] 
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1

Issue Links

• is caused by: IMPALA-11699 Some FE tests failed by NullPointerException in FileSystemUtil
• links to: Code Review