Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11806 Spark 2.0 deprecations and removals
  3. SPARK-13843

Move streaming-flume, streaming-mqtt, streaming-zeromq, streaming-akka, streaming-twitter to Spark packages

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: DStreams
    • Labels:
      None

      Description

      Currently there are a few sub-projects, each for integrating with different external sources for Streaming. Now that we have better ability to include external libraries (Spark packages) and with Spark 2.0 coming up, we can move the following projects out of Spark to https://github.com/spark-packages

      • streaming-flume
      • streaming-akka
      • streaming-mqtt
      • streaming-zeromq
      • streaming-twitter

      They are just some ancillary packages and considering the overhead of maintenance, running tests and PR failures, it's better to maintain them out of Spark. In addition, these projects can have their different release cycles and we can release them faster.

        Issue Links

          Activity

          Hide
          apachespark Apache Spark added a comment -

          User 'zsxwing' has created a pull request for this issue:
          https://github.com/apache/spark/pull/11672

          Show
          apachespark Apache Spark added a comment - User 'zsxwing' has created a pull request for this issue: https://github.com/apache/spark/pull/11672
          Hide
          srowen Sean Owen added a comment -

          Where did this code end up, by the way? we probably need a link to it.

          Show
          srowen Sean Owen added a comment - Where did this code end up, by the way? we probably need a link to it.
          Hide
          zsxwing Shixiong Zhu added a comment -

          They are in https://github.com/spark-packages

          I'm working on moving docs to them and will add their links to Spark docs.

          Show
          zsxwing Shixiong Zhu added a comment - They are in https://github.com/spark-packages I'm working on moving docs to them and will add their links to Spark docs.
          Hide
          srowen Sean Owen added a comment -

          That's great – what's the theory about who maintains them? they're no longer part of Spark itself. I'm guessing it's whoever wants to contribute and/or asks nicely to be a contributor can just do so?

          Show
          srowen Sean Owen added a comment - That's great – what's the theory about who maintains them? they're no longer part of Spark itself. I'm guessing it's whoever wants to contribute and/or asks nicely to be a contributor can just do so?
          Hide
          zsxwing Shixiong Zhu added a comment -

          Sure. These projects are just like other SQL data source spark packages.

          Show
          zsxwing Shixiong Zhu added a comment - Sure. These projects are just like other SQL data source spark packages.
          Hide
          srowen Sean Owen added a comment -

          It's a little different in that it started as part of the ASF project. It has been removed from the project (OK), it's been forked and maintained by others outside the project (OK), and nobody has now less access to it (i.e. I assume any committer would be added as a project member if they wanted to). We have to make sure it's not presented from official docs as still part of Spark, and can't release it together in a way that suggests it's official (I assume we won't).

          We also have to be careful this doesn't add up to appearing to take a part of a community project "private". These modules are so ancillary that I can't imagine it's controversial. It's going to be more of an issue for the Kafka integration (which we are discussing on the other JIRA)

          Show
          srowen Sean Owen added a comment - It's a little different in that it started as part of the ASF project. It has been removed from the project (OK), it's been forked and maintained by others outside the project (OK), and nobody has now less access to it (i.e. I assume any committer would be added as a project member if they wanted to). We have to make sure it's not presented from official docs as still part of Spark, and can't release it together in a way that suggests it's official (I assume we won't). We also have to be careful this doesn't add up to appearing to take a part of a community project "private". These modules are so ancillary that I can't imagine it's controversial. It's going to be more of an issue for the Kafka integration (which we are discussing on the other JIRA)
          Hide
          zsxwing Shixiong Zhu added a comment -

          They are all public. And definitely, any contributions are welcome. For now I'm just helping set up these projects. We will add people if they want to maintain these projects.

          For docs, I'm going to go through the official docs and remove all related to these projects.

          Show
          zsxwing Shixiong Zhu added a comment - They are all public. And definitely, any contributions are welcome. For now I'm just helping set up these projects. We will add people if they want to maintain these projects. For docs, I'm going to go through the official docs and remove all related to these projects.
          Hide
          proflin Liwei Lin(Inactive) added a comment - - edited

          hi Shixiong Zhu, we didn't move streaming-kinesis (which is also under external) out – is this left out on purpose or should we also move that out? Thanks!

          Show
          proflin Liwei Lin(Inactive) added a comment - - edited hi Shixiong Zhu , we didn't move streaming-kinesis (which is also under external) out – is this left out on purpose or should we also move that out? Thanks!
          Hide
          srowen Sean Owen added a comment -

          Yeah same question about Ganglia support

          Show
          srowen Sean Owen added a comment - Yeah same question about Ganglia support
          Hide
          rxin Reynold Xin added a comment -

          BTW I think we should give all committers write access to the github repo. I worry much more about these repos not having enough maintainers rather than the opposite of too many companies wanting to "own" them.

          Show
          rxin Reynold Xin added a comment - BTW I think we should give all committers write access to the github repo. I worry much more about these repos not having enough maintainers rather than the opposite of too many companies wanting to "own" them.
          Hide
          chrismattmann Chris A. Mattmann added a comment -

          Hi - what is being done about concerns from at least one committer here e.g., based on recent dev list discussion about moving code out of the ASF and into a Github repo without more discussion from the PMC?

          Show
          chrismattmann Chris A. Mattmann added a comment - Hi - what is being done about concerns from at least one committer here e.g., based on recent dev list discussion about moving code out of the ASF and into a Github repo without more discussion from the PMC?
          Show
          srowen Sean Owen added a comment - This has forked to dev@: http://apache-spark-developers-list.1001551.n3.nabble.com/SPARK-13843-and-future-of-streaming-backends-td16711.html
          Hide
          srowen Sean Owen added a comment -

          ... and per http://apache-spark-developers-list.1001551.n3.nabble.com/SPARK-13843-Next-steps-td16783.html it looks like the resolution is: streaming-flume comes back in; everything else stays as is: SPARK-14073

          Show
          srowen Sean Owen added a comment - ... and per http://apache-spark-developers-list.1001551.n3.nabble.com/SPARK-13843-Next-steps-td16783.html it looks like the resolution is: streaming-flume comes back in; everything else stays as is: SPARK-14073

            People

            • Assignee:
              zsxwing Shixiong Zhu
              Reporter:
              zsxwing Shixiong Zhu
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development