[HADOOP-17812] NPE in S3AInputStream read() after failure to reconnect to store

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.2, 3.3.1
• Fix Version/s: 3.4.0, 3.3.2
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Bobby Wang
• Reporter: Bobby Wang
• Created: 22/Jul/21 04:04
• Updated: 23/Aug/21 14:24
• Resolved: 30/Jul/21 22:06

Description

when reading from S3a storage, SSLException (which extends IOException) happens, which will trigger onReadFailure.

onReadFailure calls "reopen". it will first close the original wrappedStream and set wrappedStream = null, and then it will try to re-get wrappedStream. But what if the previous code obtaining S3Object throw exception, then "wrappedStream" will be null.

And the retry mechanism may re-execute the wrappedStream.read and cause NPE.

 

For more details, please refer to https://github.com/NVIDIA/spark-rapids/issues/2915

Issue Links

• depends upon: HADOOP-17764 S3AInputStream read does not re-open the input stream on the second read retry attempt
• Testing discovered: HADOOP-17824 ITestCustomSigner fails with NPE against private endpoint
• links to: GitHub Pull Request #3222
• links to: GitHub Pull Request #3251
• links to: https://github.com/NVIDIA/spark-rapids/issues/2915