[HADOOP-17934] NullPointerException when no HTTP response set on AbfsRestOperation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.3.2
• Component/s: fs/azure
• Labels: pull-request-available
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 23/Sep/21 20:45
• Updated: 07/Jan/22 14:15
• Resolved: 30/Sep/21 12:40

Description

Seen when running HBase 2.2 on top of ABFS with Hadoop 3.1ish:

Caused by: java.lang.NullPointerException
        at org.apache.hadoop.fs.azurebfs.services.AbfsClient.renameIdempotencyCheckOp(AbfsClient.java:382)
        at org.apache.hadoop.fs.azurebfs.services.AbfsClient.renamePath(AbfsClient.java:348)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.rename(AzureBlobFileSystemStore.java:722)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.rename(AzureBlobFileSystem.java:327)
        at org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:249)
        at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.rename(HRegionFileSystem.java:1115) 

Digging in, it looks like the AbfsHttpOperation inside of AbfsRestOperation may sometimes be null, but AbfsClient will try to unwrap it (and read the status code from the HTTP call). I'm not sure why we sometimes get the null HttpOperation but it seems pretty straightforward to not get the NPE.

HBase got wedged after this, but I'm not sure if it's because of this NPE or (perhaps) we weren't getting any responses from ABFS itself (i.e. there was some ABFS outage/unavailability or the node itself couldn't talk to ABFS).

Issue Links

• causes: HADOOP-17989 ITestAzureBlobFileSystemDelete failing "Operations has null HTTP response"
• is related to: HADOOP-18012 ABFS: Enable config controlled ETag check for Rename idempotency
• is related to: HADOOP-18002 abfs rename idempotency broken -remove recovery
• links to: GitHub Pull Request #3477