[HADOOP-14478] Optimize NativeAzureFsInputStream for positional reads

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Component/s: fs/azure
• Target Version/s: 2.9.0, 3.0.0-alpha4
• Hadoop Flags: Reviewed
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 02/Jun/17 06:29
• Updated: 16/May/19 21:40
• Resolved: 05/Jun/17 23:05

Description

Azure's BlobbInputStream internally buffers 4 MB of data irrespective of the data length requested for. This would be beneficial for sequential reads. However, for positional reads (seek to specific location, read x number of bytes, seek back to original location) this may not be beneficial and might even download lot more data which are not used later.

It would be good to override readFully(long position, byte[] buffer, int offset, int length) for NativeAzureFsInputStream and make use of mark(readLimit) as a hint to Azure's BlobInputStream.

BlobInputStream reference: https://github.com/Azure/azure-storage-java/blob/master/microsoft-azure-storage/src/com/microsoft/azure/storage/blob/BlobInputStream.java#L448

BlobInputStream can consider this as a hint later to determine the amount of data to be read ahead. Changes to BlobInputStream would not be addressed in this JIRA.

Issue Links

• breaks: HADOOP-14500 Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails
• contains: HADOOP-14490 Upgrade azure-storage sdk version >5.4.0
• is depended upon by: HADOOP-14552 Über-jira: WASB client phase II: performance and testing
• is related to: HADOOP-14473 Optimize NativeAzureFileSystem::seek for forward seeks
• is related to: HADOOP-14552 Über-jira: WASB client phase II: performance and testing
• relates to: HADOOP-16317 ABFS: improve random read performance