Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.4.0
    • Component/s: General
    • Labels:
      None
    1. hue.png
      53 kB
      Roman Shaposhnik
    2. pig-0.10-ambrose-patches.tar.gz
      6 kB
      Andrew Purtell
    3. BIGTOP-603.patch.txt
      4 kB
      Roman Shaposhnik

      Activity

      Roman Shaposhnik made changes -
      Status Resolved [ 5 ] Closed [ 6 ]
      Roman Shaposhnik made changes -
      Status Patch Available [ 10002 ] Resolved [ 5 ]
      Resolution Fixed [ 1 ]
      Hide
      Roman Shaposhnik added a comment -

      Lets all, please, migrate the Ambrose discussion to: BIGTOP-615

      I'm really looking forward to seeing Ambrose in Bigtop and even more so in Hue.

      Show
      Roman Shaposhnik added a comment - Lets all, please, migrate the Ambrose discussion to: BIGTOP-615 I'm really looking forward to seeing Ambrose in Bigtop and even more so in Hue.
      Hide
      Bill Graham added a comment -

      +1 @Roman, good call.

      Show
      Bill Graham added a comment - +1 @Roman, good call.
      Hide
      Roman Shaposhnik added a comment -

      @Bill, let me do two things at this point:

      1. finish up the Hue integration into trunk
      2. split this discussion into a separate JIRA (dedicated to Ambrose integration)

      I can provide you with detailed notes on Pig integration over there (and we can continue talking about Ambrose integration). Deal?

      Show
      Roman Shaposhnik added a comment - @Bill, let me do two things at this point: finish up the Hue integration into trunk split this discussion into a separate JIRA (dedicated to Ambrose integration) I can provide you with detailed notes on Pig integration over there (and we can continue talking about Ambrose integration). Deal?
      Hide
      Bill Graham added a comment -

      @Roman, interesting. In that case Ambrose should be able to integrate as-is, with some work to assure we don't have port collisions in a multi-user environment.

      When I looked at Hue I didn't see any signs of Pig integration. Is that code in the Hue github repos? Having support for running Pig jobs in Hue would be a big win. Can you provide some pointers to the Hue Pig code so we can experiment with it?

      Alas, apologies for highjacking this JIRA with the Ambrose/Hue talk.

      Show
      Bill Graham added a comment - @Roman, interesting. In that case Ambrose should be able to integrate as-is, with some work to assure we don't have port collisions in a multi-user environment. When I looked at Hue I didn't see any signs of Pig integration. Is that code in the Hue github repos? Having support for running Pig jobs in Hue would be a big win. Can you provide some pointers to the Hue Pig code so we can experiment with it? Alas, apologies for highjacking this JIRA with the Ambrose/Hue talk.
      Roman Shaposhnik made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Roman Shaposhnik added a comment -

      @Bill,

      Currently Ambrose only runs embedded in the Pig client VM, so it would only work with an embedded Hue instance somehow.

      Actually, for the current Hue model wrt. Pig this will work out quite nicely – no embedded Hue required. Currently, Hue starts Pig on the client machine and provides a terminal-like interface to the Pig's shell for the user to interface with. The way I see this is that once we integrate Ambrose the chain of events will be similar, but with a section of the screen real-estate dedicated to rendering the UI that Ambrose provides (and perhaps some proxying).

      Does this make sense?

      Show
      Roman Shaposhnik added a comment - @Bill, Currently Ambrose only runs embedded in the Pig client VM, so it would only work with an embedded Hue instance somehow. Actually, for the current Hue model wrt. Pig this will work out quite nicely – no embedded Hue required. Currently, Hue starts Pig on the client machine and provides a terminal-like interface to the Pig's shell for the user to interface with. The way I see this is that once we integrate Ambrose the chain of events will be similar, but with a section of the screen real-estate dedicated to rendering the UI that Ambrose provides (and perhaps some proxying). Does this make sense?
      Hide
      Bill Graham added a comment -

      @Roman, thanks for the reminder about Hue. Hue could be useful for us and I'll give it a look to see if it would work well with our use cases.

      Currently Ambrose only runs embedded in the Pig client VM, so it would only work with an embedded Hue instance somehow. We have plans to develop Ambrose into a standalone system that deals with job stats collection and the visualization UI. I think it would make the most sense to tackle Hue integration at that point. Let me know if you have other ideas on that part.

      Show
      Bill Graham added a comment - @Roman, thanks for the reminder about Hue. Hue could be useful for us and I'll give it a look to see if it would work well with our use cases. Currently Ambrose only runs embedded in the Pig client VM, so it would only work with an embedded Hue instance somehow. We have plans to develop Ambrose into a standalone system that deals with job stats collection and the visualization UI. I think it would make the most sense to tackle Hue integration at that point. Let me know if you have other ideas on that part.
      Hide
      Prashant Kommireddi added a comment -

      That's great, thanks Roman. Based on http://archive.cloudera.com/cdh/3/hue/manual.html#_hue_installation_instructions I was under the impression Hue goes ONLY with CDH.

      I agree Ambrose is a good candidate, it does make visualization of Pig job runs so much easier.

      Show
      Prashant Kommireddi added a comment - That's great, thanks Roman. Based on http://archive.cloudera.com/cdh/3/hue/manual.html#_hue_installation_instructions I was under the impression Hue goes ONLY with CDH. I agree Ambrose is a good candidate, it does make visualization of Pig job runs so much easier.
      Hide
      Roman Shaposhnik added a comment -

      @Prashant, there's nothing really that ties Hue to CDH, except for the fact that historically CDH has been the only distro that shipped it with Hadoop. The entire point of integrating Hue with Bigtop is so that there's a 100% free and community-driven Hadoop distribution that includes it as well. Of course, aside from the integration issues that Bigtop is solving for all the projects, there's nothing that prevents anybody from using Hue with their favorite deployment of Hadoop.

      You can track Bigtop's progress wrt. Hue over here: BIGTOP-527 and support for it is definitely scheduled to appear in Bigtop 0.4.0 release.

      Once that happens our hope is that Hue will get to be used as a platform for writing any kind of UI tools to interact with the Hadoop cluster. Ambrose is a perfect candidate to be added.

      Show
      Roman Shaposhnik added a comment - @Prashant, there's nothing really that ties Hue to CDH, except for the fact that historically CDH has been the only distro that shipped it with Hadoop. The entire point of integrating Hue with Bigtop is so that there's a 100% free and community-driven Hadoop distribution that includes it as well. Of course, aside from the integration issues that Bigtop is solving for all the projects, there's nothing that prevents anybody from using Hue with their favorite deployment of Hadoop. You can track Bigtop's progress wrt. Hue over here: BIGTOP-527 and support for it is definitely scheduled to appear in Bigtop 0.4.0 release. Once that happens our hope is that Hue will get to be used as a platform for writing any kind of UI tools to interact with the Hadoop cluster. Ambrose is a perfect candidate to be added.
      Hide
      Prashant Kommireddi added a comment -

      Hi Roman, isn't Hue tied to CDH?

      Show
      Prashant Kommireddi added a comment - Hi Roman, isn't Hue tied to CDH?
      Roman Shaposhnik made changes -
      Attachment hue.png [ 12530593 ]
      Hide
      Roman Shaposhnik added a comment -

      Hi Bill! I played with Ambrose and I like it very much. Especially since we don't really have any visualization tools for Pig jobs in Bigtop. Now, given that Bigtop is now embracing Hue as the default Hadoop browser based UI (https://github.com/cloudera/hue) I would be very interested in integrating Ambrose into Hue (essentially treating it as a Hue-based Pig plugin the same way Beeswax is a Hive plugin).

      I would be curious to know your thoughts on Hue integration and in general whether you find a Hadoop UI like Hue to be interesting for the type of use cases you're dealing with.

      P.S. I'm attaching a screenshot of my browser window running Hue. You can see Hue plugins at the top left corner. I would imagine Ambrose to be one of those.

      Show
      Roman Shaposhnik added a comment - Hi Bill! I played with Ambrose and I like it very much. Especially since we don't really have any visualization tools for Pig jobs in Bigtop. Now, given that Bigtop is now embracing Hue as the default Hadoop browser based UI ( https://github.com/cloudera/hue ) I would be very interested in integrating Ambrose into Hue (essentially treating it as a Hue-based Pig plugin the same way Beeswax is a Hive plugin). I would be curious to know your thoughts on Hue integration and in general whether you find a Hadoop UI like Hue to be interesting for the type of use cases you're dealing with. P.S. I'm attaching a screenshot of my browser window running Hue. You can see Hue plugins at the top left corner. I would imagine Ambrose to be one of those.
      Hide
      Bill Graham added a comment -

      Thanks Prashant for roping me into the discussion.

      Yes, Ambrose relies on the unreleased Pig 0.11, or the patches that Andrew references.

      Ambrose has only been open sourced for about a month has been well received. We're in the early stages of the project but we're moving fast on it. We're still working on getting our first release published to maven, which should happen soon. I think it would be great to have it included in Bigtop and would be glad to help out.

      That said, I'd love to hear others take on how useful they're finding it.

      Show
      Bill Graham added a comment - Thanks Prashant for roping me into the discussion. Yes, Ambrose relies on the unreleased Pig 0.11, or the patches that Andrew references. Ambrose has only been open sourced for about a month has been well received. We're in the early stages of the project but we're moving fast on it. We're still working on getting our first release published to maven, which should happen soon. I think it would be great to have it included in Bigtop and would be glad to help out. That said, I'd love to hear others take on how useful they're finding it.
      Hide
      Prashant Kommireddi added a comment -

      Bill Graham worked on Ambrose, may be we can ask for his comments on this too. I will forward this JIRA to him.

      Show
      Prashant Kommireddi added a comment - Bill Graham worked on Ambrose, may be we can ask for his comments on this too. I will forward this JIRA to him.
      Hide
      Roman Shaposhnik added a comment -

      Andrew, looking at the patches I see that all of them are now in Pig's trunk. It seems we should have no problem switching to that version once Pig 11 gets released.

      As a related question – what's your experience with Ambrose? Would it make sense to pull it into Bigtop somehow or is it still rather niche?

      Show
      Roman Shaposhnik added a comment - Andrew, looking at the patches I see that all of them are now in Pig's trunk. It seems we should have no problem switching to that version once Pig 11 gets released. As a related question – what's your experience with Ambrose? Would it make sense to pull it into Bigtop somehow or is it still rather niche?
      Hide
      Bruno Mahé added a comment -

      +1 to Roman's patch.

      Andrew> I would recommend to open another ticket for you case so we can see how to proceed.

      Show
      Bruno Mahé added a comment - +1 to Roman's patch. Andrew> I would recommend to open another ticket for you case so we can see how to proceed.
      Hide
      Konstantin Boudnik added a comment -

      Sounds like a good idea

      Show
      Konstantin Boudnik added a comment - Sounds like a good idea
      Hide
      Roman Shaposhnik added a comment -

      BOM is just an extra file that now gets included in SDEB/SRPM with a complete Bill Of Material for a particular build of Bigtop. The idea is that individual do-component-build scripts will use this information to transparently adjust the versions of dependencies they pull. That way we don't have to keep adjusting the versions in every single do-component-build when we bump something like hadoop or zookeeper up.

      Show
      Roman Shaposhnik added a comment - BOM is just an extra file that now gets included in SDEB/SRPM with a complete Bill Of Material for a particular build of Bigtop. The idea is that individual do-component-build scripts will use this information to transparently adjust the versions of dependencies they pull. That way we don't have to keep adjusting the versions in every single do-component-build when we bump something like hadoop or zookeeper up.
      Hide
      Bruno Mahé added a comment - - edited

      Roman> What problem are you trying to solve with this BOM? Is it meant to be only included in the srpm?

      Andew> We cannot release Apache Bigtop with patched components. But we already have introduced patches temporary for testing and while working with upstream on their inclusion in future releases. But in any case we can always have that in a branch.

      Show
      Bruno Mahé added a comment - - edited Roman> What problem are you trying to solve with this BOM? Is it meant to be only included in the srpm? Andew> We cannot release Apache Bigtop with patched components. But we already have introduced patches temporary for testing and while working with upstream on their inclusion in future releases. But in any case we can always have that in a branch.
      Andrew Purtell made changes -
      Attachment pig-0.10-ambrose-patches.tar.gz [ 12530227 ]
      Hide
      Andrew Purtell added a comment -

      We have a private Bigtop with Pig 0.10.0 and four patches backported from trunk for supporting Ambrose real time Pig job progress and dependency visualization. It might work to ask upstream to integrate these patches into 0.10 branch, but also the patches are quite self contained. I attach them here if you wish to consider some option involving them (or not).

      Show
      Andrew Purtell added a comment - We have a private Bigtop with Pig 0.10.0 and four patches backported from trunk for supporting Ambrose real time Pig job progress and dependency visualization. It might work to ask upstream to integrate these patches into 0.10 branch, but also the patches are quite self contained. I attach them here if you wish to consider some option involving them (or not).
      Roman Shaposhnik made changes -
      Field Original Value New Value
      Attachment BIGTOP-603.patch.txt [ 12530217 ]
      Hide
      Roman Shaposhnik added a comment -

      I also took the liberty to introduce bigtop.bom. Which is a shell-friendly key-value settings of all the versioning information for a Bigtop build

      Show
      Roman Shaposhnik added a comment - I also took the liberty to introduce bigtop.bom. Which is a shell-friendly key-value settings of all the versioning information for a Bigtop build
      Roman Shaposhnik created issue -

        People

        • Assignee:
          Roman Shaposhnik
          Reporter:
          Roman Shaposhnik
        • Votes:
          0 Vote for this issue
          Watchers:
          8 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development