Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3724

Use blobstore dir modtime to avoid update lookups by HDFSBlobstore

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None

      Description

      We have multiple storm clusters with 100's of supervisors polling for blob updates.  This causes high load on our Hadoop namenodes that are also used by multiple other clusters.

       

      An improvement would be for the AsyncLocalizer to check the remote blobstore last mod time once and then skip checking each individual blob if it was already checked for the same mod time.

       

        Attachments

          Activity

            People

            • Assignee:
              agresch Aaron Gresch
              Reporter:
              agresch Aaron Gresch
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4.5h
                4.5h