Details

    • Type: Wish
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: runner-ideas
    • Labels:
      None

      Description

      Gathering place for interest in a Storm runner for Beam.

        Issue Links

          Activity

          Hide
          baibaichen Chang chen added a comment -

          Any progress of Storm runner? It would be great to share some design plans with the wider community.

          Thanks
          Chang

          Show
          baibaichen Chang chen added a comment - Any progress of Storm runner? It would be great to share some design plans with the wider community. Thanks Chang
          Hide
          avermeerbergen Alexandre Vermeerbergen added a comment -

          Hello,

          An Apache Storm runner for Apache Beam would be great for us who are quite used to using Storm for very fast processing, but we are too lazy to try alternative streaming systems when we have requirements which do not require same speed as Storm.

          Part of the laziness comes from the fact that, without a unique API, we would have to rewrite our code for alternative systems such as Spark, Flink, etc.

          If Apache Storm runner were already available, then we could write new code using Beam API, being confident that we could at least run it using Storm, then start checking its behavior, operational cost on other runners.

          Note: we do not use Trident, just pure Storm API.

          Best regards,
          Alexandre

          Show
          avermeerbergen Alexandre Vermeerbergen added a comment - Hello, An Apache Storm runner for Apache Beam would be great for us who are quite used to using Storm for very fast processing, but we are too lazy to try alternative streaming systems when we have requirements which do not require same speed as Storm. Part of the laziness comes from the fact that, without a unique API, we would have to rewrite our code for alternative systems such as Spark, Flink, etc. If Apache Storm runner were already available, then we could write new code using Beam API, being confident that we could at least run it using Storm, then start checking its behavior, operational cost on other runners. Note: we do not use Trident, just pure Storm API. Best regards, Alexandre
          Hide
          davor Davor Bonaci added a comment -

          I don't think we have any measurable code progress to report at this time.

          On the Beam side, we are very interested in the Storm runner. Talking with the Storm community, I know they are interested as well. I think it is just a matter of time and priorities until this becomes a reality.

          I'd be happy to help and discuss the details with potential contributors. (Note, however, that this is a significant, longer-term undertaking – you should have a deep knowledge of either Storm or Beam internals and commit to it for some amount of time.)

          Show
          davor Davor Bonaci added a comment - I don't think we have any measurable code progress to report at this time. On the Beam side, we are very interested in the Storm runner. Talking with the Storm community, I know they are interested as well. I think it is just a matter of time and priorities until this becomes a reality. I'd be happy to help and discuss the details with potential contributors. (Note, however, that this is a significant, longer-term undertaking – you should have a deep knowledge of either Storm or Beam internals and commit to it for some amount of time.)
          Hide
          mauzhang Manu Zhang added a comment -

          Here is a feature-branch on Storm that you can watch on. AFAIK, the storm community prefer to develop and put the runner at Storm's side.

          Show
          mauzhang Manu Zhang added a comment - Here is a feature-branch on Storm that you can watch on. AFAIK, the storm community prefer to develop and put the runner at Storm's side.
          Hide
          davor Davor Bonaci added a comment -

          Having a Storm runner would be great! Awesome!

          At some point (in the future), I think it is worth clarifying the pros and cons of where the runner lives. On the technical side, I think it comes down to the API stability between different pairs of components (but, of course, there are other considerations as well).

          Show
          davor Davor Bonaci added a comment - Having a Storm runner would be great! Awesome! At some point (in the future), I think it is worth clarifying the pros and cons of where the runner lives. On the technical side, I think it comes down to the API stability between different pairs of components (but, of course, there are other considerations as well).
          Hide
          LI,Xiao Xiao LI added a comment -

          Any news about Storm runner? The last commit for storm-beamhttps://github.com/apache/storm/tree/beam-runner/external/storm-beam was 8 month ago.
          Do people give up this plan?

          Show
          LI,Xiao Xiao LI added a comment - Any news about Storm runner? The last commit for storm-beam https://github.com/apache/storm/tree/beam-runner/external/storm-beam was 8 month ago. Do people give up this plan?
          Hide
          kenn Kenneth Knowles added a comment -

          The beginnings of a JStorm runner have been contributed to a feature branch: https://github.com/apache/beam/tree/jstorm-runner

          There is more code on the way to make this more of a full prototype. Perhaps this can be the start of a runner that supports both. Are you perhaps interested in contributing to this effort?

          Show
          kenn Kenneth Knowles added a comment - The beginnings of a JStorm runner have been contributed to a feature branch: https://github.com/apache/beam/tree/jstorm-runner There is more code on the way to make this more of a full prototype. Perhaps this can be the start of a runner that supports both. Are you perhaps interested in contributing to this effort?
          Hide
          peihe0@gmail.com Pei He added a comment -

          Hi Xiao LI,
          The jstorm runner is in my github repo, and we are working on merging it to Beam repo.
          https://github.com/peihe/incubator-beam/tree/jstorm-runner-wip

          If you or anyone can help investigating how to port it to the storm API, that would be great.

          Show
          peihe0@gmail.com Pei He added a comment - Hi Xiao LI, The jstorm runner is in my github repo, and we are working on merging it to Beam repo. https://github.com/peihe/incubator-beam/tree/jstorm-runner-wip If you or anyone can help investigating how to port it to the storm API, that would be great.
          Hide
          LI,Xiao Xiao LI added a comment - - edited

          It's great to hear the progress about storm runner. Our team are looking for an unified way to manage unbounded dataset processing. I will investigate the feasibility and cost for our team. Thank you all!

          Show
          LI,Xiao Xiao LI added a comment - - edited It's great to hear the progress about storm runner. Our team are looking for an unified way to manage unbounded dataset processing. I will investigate the feasibility and cost for our team. Thank you all!

            People

            • Assignee:
              sriharsha Sriharsha Chintalapani
              Reporter:
              frances Frances Perry
            • Votes:
              4 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:

                Development