[HADOOP-15478] WASB: hflush() and hsync() regression

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0, 3.0.2
• Fix Version/s: 2.10.0, 3.1.1
• Component/s: fs/azure
• Release Note: WASB: Bug fix for recent regression in hflush() and hsync().
• Assignee: Thomas Marqardt
• Reporter: Thomas Marqardt
• Created: 18/May/18 08:59
• Updated: 18/Jun/18 05:40
• Resolved: 21/May/18 11:08

Description

HADOOP-14520 introduced a regression in hflush() and hsync().  Previously, for the default case where users upload data as block blobs, these were no-ops.  Unfortunately, HADOOP-14520 accidentally implemented hflush() and hsync() by default, so any data buffered in the stream is immediately uploaded to storage.  This new behavior is undesirable, because block blobs have a limit of 50,000 blocks.  Spark users are now seeing failures due to exceeding the block limit, since Spark frequently invokes hflush().