Uploaded image for project: 'Giraph'
  1. Giraph
  2. GIRAPH-568

Giraph on YARN will need a WebUI to display job stats, Yammer metrics, whatever

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      In YARN, the Client is the driver program when you run a job at the commmand line. This lauches the application master, which is like an uber-master that manages the job lifecycle for all the Giraph worker/master tasks that actually run the BSP job.

      The Application Master can register an RPC Port and a Tracking URL with the YARN system (ResourceManager) which will be published on the YARN cluster WebUI in case folks running a Giraph job want to see detailed formatted web info such as Hadoop has. Previously we have hijacked Hadoop's counters and web ui. Now, we can start to think fresh about how to read logs, view job and node status, memory use, disk spills, Yammer metrics, whatever.

      Someone could get very creative with this. If someone is feeling up to it, I can show you where the YARN bits are you will want to interface with. The rest can really go any way you want it to.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              initialcontext Eli Reisman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: