Details

    • Type: Sub-task
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: storm-core

      Description

      Details in these documents:

      1) Redesign of the messaging subsystem
      https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing
      This doc discusses the new design for the messaging system. Plus some of the optimizations being made.

      2) Choosing a high performance messaging queue:
      https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing
      This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues.

      3) Backpressure Model
      https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing
      Describes the Backpressure model integrated into the new messaging subsystem.

        Issue Links

          Activity

          Hide
          roshan_naik Roshan Naik added a comment -

          Since the PR webpage has become terribly slow due to the large number of comments, moving the non-code-review type conversation to the JIRA.

          The PR is ready to move forward with additional reviews and testing by anyone interested in test driving their topologies on this (much appreciated!).

          • I believe all the key issues raised so far should be addressed, please take a look. Also includes fixes to issues discovered during testing and perf runs.
          • Have added a new design doc detailing the BackPressure model. That is the part that has undergone the most change off late.
          • Based on my observations from perf runs and prior feedback on the PR, the new defaults have been tweaked to make it easy for existing workloads to transition to this with minimal or no tweaking but still get good perf.
          • My colleague is in the process of running some perf numbers comparing master vs the latest 2306. Will share them soon.

          cc: Robert Joseph Evans

          Show
          roshan_naik Roshan Naik added a comment - Since the PR webpage has become terribly slow due to the large number of comments, moving the non-code-review type conversation to the JIRA. The PR is ready to move forward with additional reviews and testing by anyone interested in test driving their topologies on this (much appreciated!). I believe all the key issues raised so far should be addressed, please take a look. Also includes fixes to issues discovered during testing and perf runs. Have added a new design doc detailing the BackPressure model. That is the part that has undergone the most change off late. Based on my observations from perf runs and prior feedback on the PR, the new defaults have been tweaked to make it easy for existing workloads to transition to this with minimal or no tweaking but still get good perf. My colleague is in the process of running some perf numbers comparing master vs the latest 2306. Will share them soon. cc: Robert Joseph Evans

            People

            • Assignee:
              roshan_naik Roshan Naik
              Reporter:
              roshan_naik Roshan Naik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 33h
                33h

                  Development