Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0
    • timelineserver

    Description

      As planned, we need to try out the new YARN web UI framework and implement timeline v2 web UI on top of it. This JIRA proposes to build the basic active flow and application lists of the timeline data. We can add more content after we get used to this framework.

      Attachments

        1. Screen Shot 2016-02-24 at 15.57.38.png
          268 kB
          Li Lu
        2. Screen Shot 2016-02-24 at 15.57.53.png
          57 kB
          Li Lu
        3. Screen Shot 2016-02-24 at 15.58.08.png
          63 kB
          Li Lu
        4. Screen Shot 2016-02-24 at 15.58.26.png
          365 kB
          Li Lu
        5. YARN-4097-bugfix.patch
          46 kB
          Li Lu
        6. YARN-4097-POC.patch
          73 kB
          Li Lu

        Issue Links

          Activity

            gtcarrera9 Li Lu added a comment -

            We may want to make use of the latest progress in YARN-3368 and try it here. This can be the starting point of our experimental UI.

            gtcarrera9 Li Lu added a comment - We may want to make use of the latest progress in YARN-3368 and try it here. This can be the starting point of our experimental UI.
            vrushalic Vrushali C added a comment -

            Hi gtCarrera Wondering if this jira should be reassigned to someone who is working on the UI PoC? Or are you working on it? Would be great to see some interim/in progress screenshots or anything else that you can share.

            vrushalic Vrushali C added a comment - Hi gtCarrera Wondering if this jira should be reassigned to someone who is working on the UI PoC? Or are you working on it? Would be great to see some interim/in progress screenshots or anything else that you can share.
            gtcarrera9 Li Lu added a comment -

            Yes I'm working on this right now. Currently I've added a page for listing all past active flows in the system, and flowruns inside a flow. I think it's a good time for some discussion about the progress and next step plans.

            gtcarrera9 Li Lu added a comment - Yes I'm working on this right now. Currently I've added a page for listing all past active flows in the system, and flowruns inside a flow. I think it's a good time for some discussion about the progress and next step plans.
            gtcarrera9 Li Lu added a comment -

            OK here are some screenshots for the current POC we have for the ATS related web UI pages. Note that we're affected by YARN-4700 in the flow activity list (there are multiple items for the same flow due to different cluster ids). For flow and flowrun page now the version is very immature, but I left some possibility to further integrate more data from aggregations. The application page in the new YARN UI currently only reads data from the RM. Right now I simply link to the page for future integrations.

            For the near term next steps, we may want:

            1. Provide a "dashboard" in the flow activity page, summarizing flow activities.
            2. Integrate flowrun/flow level aggregation data to flow and flowrun page.
            3. Integrate ATS v2 web services to the application page, so that once an application finished, data can be read from the timeline server rather than the RM. We may want to extend this practice to attempt and container page.
            gtcarrera9 Li Lu added a comment - OK here are some screenshots for the current POC we have for the ATS related web UI pages. Note that we're affected by YARN-4700 in the flow activity list (there are multiple items for the same flow due to different cluster ids). For flow and flowrun page now the version is very immature, but I left some possibility to further integrate more data from aggregations. The application page in the new YARN UI currently only reads data from the RM. Right now I simply link to the page for future integrations. For the near term next steps, we may want: Provide a "dashboard" in the flow activity page, summarizing flow activities. Integrate flowrun/flow level aggregation data to flow and flowrun page. Integrate ATS v2 web services to the application page, so that once an application finished, data can be read from the timeline server rather than the RM. We may want to extend this practice to attempt and container page.
            gtcarrera9 Li Lu added a comment -

            BTW, I haven't fine-tune the styles of our current pages. More decorations will be very helpful.

            gtcarrera9 Li Lu added a comment - BTW, I haven't fine-tune the styles of our current pages. More decorations will be very helpful.

            I left a comment here proposing that we decouple YARN-2928 merge and the UI dependency for the merge - please see my comment there and share your thoughts. We still need to plan on how we deliver an end-to-end UI that showcases YTS V2, but we should decouple its timeline from the merge plans.

            vinodkv Vinod Kumar Vavilapalli added a comment - I left a comment here proposing that we decouple YARN-2928 merge and the UI dependency for the merge - please see my comment there and share your thoughts. We still need to plan on how we deliver an end-to-end UI that showcases YTS V2, but we should decouple its timeline from the merge plans.
            gtcarrera9 Li Lu added a comment -

            I've rebased the POC patch to the latest YARN-3368 branch. While rebasing and cleaning up the code, I noticed one bug is that when I use rsvp.hash with store.query, the result of the route's query will not be updated unless a refresh. I'm not sure what happened there, and I'll continue working on integrating app and container pages with timeline v2. At the same time, it would be great if someone could give me some hint on that... Thanks!

            BTW I'm using em-table, so cc/Sreenath.

            gtcarrera9 Li Lu added a comment - I've rebased the POC patch to the latest YARN-3368 branch. While rebasing and cleaning up the code, I noticed one bug is that when I use rsvp.hash with store.query, the result of the route's query will not be updated unless a refresh. I'm not sure what happened there, and I'll continue working on integrating app and container pages with timeline v2. At the same time, it would be great if someone could give me some hint on that... Thanks! BTW I'm using em-table, so cc/ Sreenath .

            gtCarrera9
            I would like to help on the Ember side. Please guide me to the code where you are facing issue with ember data.

            Sreenath Sreenath Somarajapuram added a comment - gtCarrera9 I would like to help on the Ember side. Please guide me to the code where you are facing issue with ember data.
            gtcarrera9 Li Lu added a comment -

            Thanks Sreenath for the help! Right now the code is not yet checked in, but I'll send you a copy so that you may want to take a look at.

            gtcarrera9 Li Lu added a comment - Thanks Sreenath for the help! Right now the code is not yet checked in, but I'll send you a copy so that you may want to take a look at.
            gtcarrera9 Li Lu added a comment -

            OK here's the current patch (applies to branch YARN-3368 which sets up the new UI of YARN). The symptom is that in the flow run page, related application information will not be updated unless you refresh the page. The data for those applications is obtained in routes/yarn-flowrun through:

            return Ember.RSVP.hash({
                  flowrun: this.store.findRecord('yarn-flowrun', params.flowrun_uid),
                  apps: this.store.query('yarn-app-flowrun', {flowrunUid: params.flowrun_uid})
                });
            

            Not sure why each time the flowrun is updated accordingly, but apps is not...

            gtcarrera9 Li Lu added a comment - OK here's the current patch (applies to branch YARN-3368 which sets up the new UI of YARN). The symptom is that in the flow run page, related application information will not be updated unless you refresh the page. The data for those applications is obtained in routes/yarn-flowrun through: return Ember.RSVP.hash({ flowrun: this .store.findRecord( 'yarn-flowrun' , params.flowrun_uid), apps: this .store.query( 'yarn-app-flowrun' , {flowrunUid: params.flowrun_uid}) }); Not sure why each time the flowrun is updated accordingly, but apps is not...
            varun_saxena Varun Saxena added a comment - - edited

            gtCarrera9, in the patch above, there is no link to flow activities in the top level tab.
            I have to access yarn-flow-activity URL(to access flows) directly via browser address bar. So that can be fixed

            By the way, I was trying out sunburst diagrams to display flow run aggregations. Will be doing that on top of this patch.

            varun_saxena Varun Saxena added a comment - - edited gtCarrera9 , in the patch above, there is no link to flow activities in the top level tab. I have to access yarn-flow-activity URL(to access flows) directly via browser address bar. So that can be fixed By the way, I was trying out sunburst diagrams to display flow run aggregations. Will be doing that on top of this patch.
            gtcarrera9 Li Lu added a comment -

            Hi varun_saxena, maybe the patch does not quite apply to the latest YARN-3368 branch? I can post a new one if needed...

            gtcarrera9 Li Lu added a comment - Hi varun_saxena , maybe the patch does not quite apply to the latest YARN-3368 branch? I can post a new one if needed...
            varun_saxena Varun Saxena added a comment - - edited

            The patch applies. I meant that we have a navigation bar on top of the UI page which has links to Cluster Overview, Applications, Nodes, etc. We do not have a similar link for Flows there.
            It was there in an earlier patch you had shared once offline. It seems to have been added in application controller in your patch. It needs to be added in templates/application.hbs instead.

            It was there in controller earlier but it seems to have changed since then.

            varun_saxena Varun Saxena added a comment - - edited The patch applies. I meant that we have a navigation bar on top of the UI page which has links to Cluster Overview, Applications, Nodes, etc. We do not have a similar link for Flows there. It was there in an earlier patch you had shared once offline. It seems to have been added in application controller in your patch. It needs to be added in templates/application.hbs instead. It was there in controller earlier but it seems to have changed since then.
            gtcarrera9 Li Lu added a comment -

            OK got your point. Yes that was not included. Can we sync up this work with YARN-3368's progress? Seems like they're also doing some refactoring on application.hbs in the UI branch.

            gtcarrera9 Li Lu added a comment - OK got your point. Yes that was not included. Can we sync up this work with YARN-3368 's progress? Seems like they're also doing some refactoring on application.hbs in the UI branch.
            varun_saxena Varun Saxena added a comment - - edited

            Can we sync up this work with YARN-3368's progress ?

            Yeah it seems there will be some refactoring. Just wanted to point out because we may want to give UI demo(ATS specific) too at upcoming ATS talk. We can fix this after refactoring work going on in YARN-3368.

            varun_saxena Varun Saxena added a comment - - edited Can we sync up this work with YARN-3368 's progress ? Yeah it seems there will be some refactoring. Just wanted to point out because we may want to give UI demo(ATS specific) too at upcoming ATS talk. We can fix this after refactoring work going on in YARN-3368 .
            gtcarrera9 Li Lu added a comment -

            Since YARN-2928 has been successfully merged into trunk, shall we carry this task over to the YARN-3368 branch? We can use the summit demo code as a starting point in YARN-3368. cc/wangda

            gtcarrera9 Li Lu added a comment - Since YARN-2928 has been successfully merged into trunk, shall we carry this task over to the YARN-3368 branch? We can use the summit demo code as a starting point in YARN-3368 . cc/ wangda
            leftnoteasy Wangda Tan added a comment -

            gtCarrera9, varun_saxena.

            We have committed all our internal changes to branch: YARN-3368, you can rebase your works on top of YARN-3368.

            leftnoteasy Wangda Tan added a comment - gtCarrera9 , varun_saxena . We have committed all our internal changes to branch: YARN-3368 , you can rebase your works on top of YARN-3368 .

            Since YARN-2928 has been successfully merged into trunk, shall we carry this task over to the YARN-3368 branch?

            +1, this way when YARN-3368 goes into trunk, we will have a fully functional UI that works across RM, NMs and ATS V2.

            vinodkv Vinod Kumar Vavilapalli added a comment - Since YARN-2928 has been successfully merged into trunk, shall we carry this task over to the YARN-3368 branch? +1, this way when YARN-3368 goes into trunk, we will have a fully functional UI that works across RM, NMs and ATS V2.
            gtcarrera9 Li Lu added a comment -

            Rebase the current POC patch with the latest YARN-3368 branch. This patch integrated varun_saxena's sunburst chart contributed in YARN-4097.

            gtcarrera9 Li Lu added a comment - Rebase the current POC patch with the latest YARN-3368 branch. This patch integrated varun_saxena 's sunburst chart contributed in YARN-4097 .
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 patch 0m 5s YARN-4097 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-4097 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820815/YARN-4097-POC.patch JIRA Issue YARN-4097 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12549/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            gsaha Gour Saha added a comment -

            Talked to gtCarrera9 and all POC work on this is complete. The POC patch has been handed over to YARN-3368. Marking this resolved.

            gsaha Gour Saha added a comment - Talked to gtCarrera9 and all POC work on this is complete. The POC patch has been handed over to YARN-3368 . Marking this resolved.

            People

              gtcarrera9 Li Lu
              gtcarrera9 Li Lu
              Votes:
              1 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: