[HADOOP-18456] NullPointerException in ObjectListingIterator's constructor

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.5
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Quanlong Huang
• Created: 19/Sep/22 02:31
• Updated: 28/Jan/24 00:09
• Resolved: 23/Sep/22 09:10

Description

We saw NullPointerExceptions in Impala's S3 tests: IMPALA-11592. It's thrown from the hadoop jar:

Caused by: java.lang.NullPointerException
        at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.<init>(Listing.java:621)
        at org.apache.hadoop.fs.s3a.Listing.createObjectListingIterator(Listing.java:163)
        at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:144)
        at org.apache.hadoop.fs.s3a.Listing.getListFilesAssumingDir(Listing.java:212)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:4790)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listFiles$37(S3AFileSystem.java:4732)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2363)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2382)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:4731)
        at org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:754)
        ... 

We are using a private build of the hadoop jar. Version: CDP 3.1.1.7.2.16.0-164
Code snipper of where the NPE throws:

604     @Retries.RetryRaw
605     ObjectListingIterator(
606         Path listPath,
607         S3ListRequest request,
608         AuditSpan span) throws IOException {
609       this.listPath = listPath;
610       this.maxKeys = listingOperationCallbacks.getMaxKeys();
611       this.request = request;
612       this.objectsPrev = null;
613       this.iostats = iostatisticsStore()
614           .withDurationTracking(OBJECT_LIST_REQUEST)
615           .withDurationTracking(OBJECT_CONTINUE_LIST_REQUEST)
616           .build();
617       this.span = span;
618       this.s3ListResultFuture = listingOperationCallbacks
619           .listObjectsAsync(request, iostats, span);
620       this.aggregator = IOStatisticsContext.getCurrentIOStatisticsContext()
621           .getAggregator();           // <---- thrown here
622     }

Issue Links

• causes: IMPALA-11592 NullPointerException from s3a.Listing.createObjectListingIterator in various S3 tests
• is caused by: HADOOP-17461 Add thread-level IOStatistics Context
• is caused by: HADOOP-18091 S3A auditing leaks memory through ThreadLocal references
• is related to: HADOOP-18767 WeakReferenceMap creation warning log doesn't format correctly
• links to: GitHub Pull Request #4909