Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Perhaps instead of constantly adding services to Whirr, we should simply have a mechanism that allows people to publish them to a repository (such as Maven's repo – note, this doesn't require people to use maven, but it does provide a place to publish) and then in Whirr, you can just say "go get this service from this repository" instead of the project itself constantly adding services and maintaining more and more "one-off" services code.

      Not sure if this is a dupe, but a quick search didn't reveal it.

        Activity

        Hide
        Andrei Savu added a comment -

        This is not a dupe - we only had this discussion before on the email list. I totally agree that we need to find a more scalable way to manage an increasing number of services. 0.8.0 is going to be OSGi friendly so in a way we can now work with services that are published by others with maven. In the Whirr core we are making use of Java SPI to discover them so even now it's possible to add more services just by dropping the jars in lib/. Is there a tool that we can use to retrieve them from maven? What approach would work best for you?

        Show
        Andrei Savu added a comment - This is not a dupe - we only had this discussion before on the email list. I totally agree that we need to find a more scalable way to manage an increasing number of services. 0.8.0 is going to be OSGi friendly so in a way we can now work with services that are published by others with maven. In the Whirr core we are making use of Java SPI to discover them so even now it's possible to add more services just by dropping the jars in lib/. Is there a tool that we can use to retrieve them from maven? What approach would work best for you?
        Hide
        Andrei Savu added a comment - - edited

        We've also considered having independent releases for core & services but we concluded this is complicated from an operational point of view and too early - the Whirr core still needs many improvements and we prefer to be forced to change the services at the same time.

        Show
        Andrei Savu added a comment - - edited We've also considered having independent releases for core & services but we concluded this is complicated from an operational point of view and too early - the Whirr core still needs many improvements and we prefer to be forced to change the services at the same time.
        Hide
        Andrei Savu added a comment - - edited

        How about building a tool that can retrieve additional services based on:
        https://docs.sonatype.org/display/AETHER/Home (thanks Tim Freeman on Twitter)

        Show
        Andrei Savu added a comment - - edited How about building a tool that can retrieve additional services based on: https://docs.sonatype.org/display/AETHER/Home (thanks Tim Freeman on Twitter)
        Show
        Andrei Savu added a comment - - edited Another suggestion: Standalone Apache Ivy http://ant.apache.org/ivy/history/latest-milestone/standalone.html https://twitter.com/#!/retronym/status/168408113748262912
        Hide
        David Arthur added a comment -

        So if there are certain services in the "blessed" repository (Apache Whirr project), and other people's services that download whirr deps from Maven, what about dependencies between unofficial services?

        Show
        David Arthur added a comment - So if there are certain services in the "blessed" repository (Apache Whirr project), and other people's services that download whirr deps from Maven, what about dependencies between unofficial services?
        Hide
        Andrei Savu added a comment - - edited

        Maybe the solution is to make good use of OSGi - it solves some of the versioning issues & allows you to easily register more services.

        Show
        Andrei Savu added a comment - - edited Maybe the solution is to make good use of OSGi - it solves some of the versioning issues & allows you to easily register more services.
        Hide
        Ioannis Canellos added a comment -

        Inside an OSGi environment this is already possible. In fact this is already tested inside the integration tests.
        The WhirrServicesTest under platforms/karaf/itests does exactly this:

        Inside a vanilla Karaf instance it registers the a descriptor which defines the artifacts required for running whirr & its services (split in logical groups: whirr, whirr-zookeeper, whirr-hadoop etc). Then it installs to the container one by one and makes sure that whirr can discover and use any of the services.

        This can be applied to any OSGi environment and not just Karaf.

        Now, lets think outside OSGi. There are two things needed. The first is dependency resolution and the second is dynamic registration.

        Dependency resolution: Even with the use of a tool like Aether or Ivy (I am only a bit familiar with the first) no one can guarantee that the resolution will succeed as it will require that the maven metadata reflect 100% the runtime requirements (its not always the case) and of course that can't prevent conflicts (maybe isolation will help here).

        Dynamic Registration: Yes dropping jars to the lib will work and its a fine approach when you are dealing with the CLI. You can even add some commands for adding new services by referencing a descriptor or something. If we are talking about whirr as a library, which is going to be used inside a java application, container etc, then it gets a bit more complicated. I think that the SPI approach will have issues if the class is not in the class loader hierarchy and that could be a problem if you are loading services isolated. Maybe commons discovery & some TCCL can do the trick.

        This is exactly why I love OSGi.

        Show
        Ioannis Canellos added a comment - Inside an OSGi environment this is already possible. In fact this is already tested inside the integration tests. The WhirrServicesTest under platforms/karaf/itests does exactly this: Inside a vanilla Karaf instance it registers the a descriptor which defines the artifacts required for running whirr & its services (split in logical groups: whirr, whirr-zookeeper, whirr-hadoop etc). Then it installs to the container one by one and makes sure that whirr can discover and use any of the services. This can be applied to any OSGi environment and not just Karaf. Now, lets think outside OSGi. There are two things needed. The first is dependency resolution and the second is dynamic registration. Dependency resolution: Even with the use of a tool like Aether or Ivy (I am only a bit familiar with the first) no one can guarantee that the resolution will succeed as it will require that the maven metadata reflect 100% the runtime requirements (its not always the case) and of course that can't prevent conflicts (maybe isolation will help here). Dynamic Registration: Yes dropping jars to the lib will work and its a fine approach when you are dealing with the CLI. You can even add some commands for adding new services by referencing a descriptor or something. If we are talking about whirr as a library, which is going to be used inside a java application, container etc, then it gets a bit more complicated. I think that the SPI approach will have issues if the class is not in the class loader hierarchy and that could be a problem if you are loading services isolated. Maybe commons discovery & some TCCL can do the trick. This is exactly why I love OSGi.
        Hide
        Andrei Savu added a comment -

        Would it be possible to build a CLI that could use a Karaf server running in background? It sounds like a good idea to me + many operations should be much faster because the compute service context is created only once.

        Show
        Andrei Savu added a comment - Would it be possible to build a CLI that could use a Karaf server running in background? It sounds like a good idea to me + many operations should be much faster because the compute service context is created only once.
        Hide
        Tom White added a comment -

        Thanks for opening this issue Grant! I agree that people should be able to write Whirr services and publish them to Maven so that anyone can run them using Whirr without installing anything extra.

        > Would it be possible to build a CLI that could use a Karaf server running in background?

        This sounds like the right approach since we want to isolate services from each other (and not just drop JARs that have been resolved using Maven/Aether into lib).

        Show
        Tom White added a comment - Thanks for opening this issue Grant! I agree that people should be able to write Whirr services and publish them to Maven so that anyone can run them using Whirr without installing anything extra. > Would it be possible to build a CLI that could use a Karaf server running in background? This sounds like the right approach since we want to isolate services from each other (and not just drop JARs that have been resolved using Maven/Aether into lib).

          People

          • Assignee:
            Unassigned
            Reporter:
            Grant Ingersoll
          • Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development