[IMPALA-11592] NullPointerException from s3a.Listing.createObjectListingIterator in various S3 tests

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Catalog
• Labels: broken-build
• Epic Color: ghx-label-12
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 18/Sep/22 01:06
• Updated: 12/Oct/22 22:27

Description

custom_cluster.test_local_catalog.TestLocalCatalogRetries.test_fetch_metadata_retry fails in a S3 build:

custom_cluster/test_local_catalog.py:317: in test_fetch_metadata_retry
    seen = self._check_metadata_retries(queries)
custom_cluster/test_local_catalog.py:293: in _check_metadata_retries
    assert failed_queries.empty(),\
E   AssertionError: Failed query count non zero: [('refresh functional.alltypes', 'ImpalaBeeswaxException:\n Query aborted:TableLoadingException: Refreshing file and block metadata for 24 paths for table functional.alltypes: failed to load 1 paths. Check the catalog server log for more details.\n\n')]
E   assert <bound method Queue.empty of <Queue.Queue instance at 0x7fe3fb710128>>()
E    +  where <bound method Queue.empty of <Queue.Queue instance at 0x7fe3fb710128>> = <Queue.Queue instance at 0x7fe3fb710128>.empty

Looking into the catalog server log, there is a NullPointerException:

E0916 21:15:27.469425 25508 ParallelFileMetadataLoader.java:171] Refreshing file and block metadata for 24 paths for table functional.alltypes encountered an error loading data for path s3a://impala-test-uswest2-2/test-warehouse/alltypes/year=2010/month=9
Java exception follows:
java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:168)
        at org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:120)
        at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:781)
        at org.apache.impala.catalog.HdfsTable.access$100(HdfsTable.java:153)
        at org.apache.impala.catalog.HdfsTable$PartitionDeltaUpdater.apply(HdfsTable.java:1534)
        at org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(HdfsTable.java:1411)
        at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1254)
        at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1179)
        at org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(CatalogServiceCatalog.java:2551)
        at org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:6158)
        at org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:287)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.<init>(Listing.java:621)
        at org.apache.hadoop.fs.s3a.Listing.createObjectListingIterator(Listing.java:163)
        at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:144)
        at org.apache.hadoop.fs.s3a.Listing.getListFilesAssumingDir(Listing.java:212)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:4790)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listFiles$37(S3AFileSystem.java:4732)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2363)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2382)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:4731)
        at org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:754)
        at org.apache.impala.common.FileSystemUtil.listStatus(FileSystemUtil.java:729)
        at org.apache.impala.catalog.FileMetadataLoader.load(FileMetadataLoader.java:190)
        at org.apache.impala.catalog.ParallelFileMetadataLoader.lambda$loadInternal$1(ParallelFileMetadataLoader.java:162)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
E0916 21:15:27.469880 25508 JniCatalog.java:295] Error in resetMetadata for REFRESH TABLE functional.alltypes issued by jenkins. Time spent: 140ms.
I0916 21:15:27.470046 25508 jni-util.cc:288] org.apache.impala.catalog.TableLoadingException: Refreshing file and block metadata for 24 paths for table functional.alltypes: failed to load 1 paths. Check the catalog server log for more details.
        at org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:185)
        at org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:120)
        at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:781)
        at org.apache.impala.catalog.HdfsTable.access$100(HdfsTable.java:153)
        at org.apache.impala.catalog.HdfsTable$PartitionDeltaUpdater.apply(HdfsTable.java:1534)
        at org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(HdfsTable.java:1411)
        at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1254)
        at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1179)
        at org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(CatalogServiceCatalog.java:2551)
        at org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:6158)
        at org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:287)

Dependent hadoop version: IMPALA_HADOOP_VERSION=3.1.1.7.2.16.0-164

Issue Links

• is caused by: HADOOP-18456 NullPointerException in ObjectListingIterator's constructor
• is related to: IMPALA-11649 Null pointer exception seen in org.apache.impala.catalog.ParallelFileMetada in impala-asf-master-core-s3 testtaLoader