[STORM-3724] Use blobstore dir modtime to avoid update lookups by HDFSBlobstore

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Assignee: Aaron Gresch
• Reporter: Aaron Gresch
• Created: 16/Dec/20 17:17
• Updated: 14/Jan/21 15:16
• Resolved: 14/Jan/21 15:16

Description

We have multiple storm clusters with 100's of supervisors polling for blob updates.  This causes high load on our Hadoop namenodes that are also used by multiple other clusters.

 

An improvement would be for the AsyncLocalizer to check the remote blobstore last mod time once and then skip checking each individual blob if it was already checked for the same mod time.