Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-1328

Evaluate/Port JStorm tuple groupings

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Storm:
      Load aware balancing in shuffle grouping

      JStorm:
      Has a "localfirst" grouping that causes tuples to be sent to the tasks in the same worker by default. But if the load of all local tasks is high, the tuples will be sent out to remote tasks.
      Improve localOrShuffle grouping from Storm. In Storm's localOrShuffle grouping the definition of "local" is local within the same worker process. i.e., if there is a bolt that the component can send to in the current worker process it will send the tuples there. If there is not one, it will do round robin between all of the instances of that bolt no matter which hosts they are on. JStorm has extended that so that other workers/JVMs on the same host are considered "local" as well, taking into account the load of the network connections on the local worker.

      We should look at combining both of these to have shuffle look at both distance and load to decide where to send a tuple, in addition the the new JStorm localfirst grouping.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fengjian 冯健
                Reporter:
                revans2 Robert Joseph Evans
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: