Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3339

Port all the AtomicReference to ConcurrentHashMap for Nimbus

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.0.0
    • None
    • storm-core
    • None

    Description

      Now for many concurrent access resource in Nimbus.java, we use AtomicReference to make them multi thread safe. The resources summarized below:

      1. heartbeatsCache
      2. schedulingStartTimeNs
      3. idToSchedStatus
      4. nodeIdToResources
      5. idToWorkerResources
      6. idToExecutors

      The 1, 4, 5 and 6 may grows huge if we have hundreds of topologies on cluster, when we update AtomicReference, actually we passed in a Function and use compareAndSet to update the whole val to the new returned by the Function, in that case, we must do a reference copy and merge the changes, which seems not necessary.

      I think the reason to use AtomicReference is a legacy from old Clojure code, we can replace them totally with ConcurrentHashMap which supported better performance.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            danny0405 Danny Chen
            danny0405 Danny Chen

            Dates

              Created:
              Updated:

              Slack

                Issue deployment