Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Existing YARN UI configuration is under Hadoop package's directory: $HADOOP_PREFIX/share/hadoop/yarn/webapps/, we should move it to $HADOOP_CONF_DIR like other configurations.

      1. 0001-YARN-5145-Run-NewUI-WithOldPort-POC.patch
        4 kB
        Sunil G
      2. newUIInOldRMWebServer.png
        147 kB
        Sunil G
      3. YARN-5145-YARN-3368.01.patch
        2 kB
        Kai Sasaki
      4. YARN-5145-YARN-3368.02.patch
        4 kB
        Sunil G
      5. YARN-5145-YARN-3368.03.patch
        5 kB
        Sunil G
      6. YARN-5145-YARN-3368.04.patch
        5 kB
        Sunil G
      7. YARN-5145-YARN-3368.05.patch
        5 kB
        Sunil G
      8. YARN-5145-YARN-3368.06.patch
        5 kB
        Sunil G

        Activity

        Hide
        lewuathe Kai Sasaki added a comment -

        Valid configurations is now all included in default-config.js and environment.js. They are build into webapp package by ember build. So configs.env is not used any more.
        If we remove the file (configs.env), config dir is not generated under webapps which was intended to be moved to $HADOOP_CONF_DIR.

        Show
        lewuathe Kai Sasaki added a comment - Valid configurations is now all included in default-config.js and environment.js. They are build into webapp package by ember build . So configs.env is not used any more. If we remove the file ( configs.env ), config dir is not generated under webapps which was intended to be moved to $HADOOP_CONF_DIR .
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        0m 52s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9baccb9
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828454/YARN-5145-YARN-3368.01.patch
        JIRA Issue YARN-5145
        Optional Tests asflicense
        uname Linux 835b92b4c31c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-3368 / 9ef8291
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13106/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 0m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:9baccb9 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828454/YARN-5145-YARN-3368.01.patch JIRA Issue YARN-5145 Optional Tests asflicense uname Linux 835b92b4c31c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-3368 / 9ef8291 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui Console output https://builds.apache.org/job/PreCommit-YARN-Build/13106/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        HI Kai Sasaki
        I didnt understand the approach here. Why are we removing configs.env

        This is the config which i used presently to run YARN web ui.. Also mentioned about same in doc files. I think what we are looking here is to have a config in $HADOOP_CONF_DIR and during RM start/re-start can place/mirror necessary conf to ember conf dir. So ember can work the way its working now, and for user, he can config like hadoop style. Thoughts?

        Show
        sunilg Sunil G added a comment - HI Kai Sasaki I didnt understand the approach here. Why are we removing configs.env This is the config which i used presently to run YARN web ui.. Also mentioned about same in doc files. I think what we are looking here is to have a config in $HADOOP_CONF_DIR and during RM start/re-start can place/mirror necessary conf to ember conf dir. So ember can work the way its working now, and for user, he can config like hadoop style. Thoughts?
        Hide
        lewuathe Kai Sasaki added a comment -

        Sunil G
        Sorry for lacking of explanation. It seems that configs.env is not used anymore because all configurations are included in default-config.js. And new YARN UI is working fine without configs.env. So we can remove this file.

        And then after removing configs.env file, there is no config directory under $HADOOP_PREFIX/share/hadoop/yarn/webapps/ because condifurations default-config.js are build into ember deploy. There is no configurations to be passed externally since they are included in ember deploy package.

        So removing configs.env is intended to
        1. Since configs.env is not used anymore, it can be removed.
        2. By removing config directory in ember deployed package, we found there is no configurations to be passed from external.

        Does it make sense?
        But if any other new configurations are introduced which should be changed from external, we need to implement a way to pass some values to deployed ember package. Do you consider of such future usage?

        Show
        lewuathe Kai Sasaki added a comment - Sunil G Sorry for lacking of explanation. It seems that configs.env is not used anymore because all configurations are included in default-config.js . And new YARN UI is working fine without configs.env . So we can remove this file. And then after removing configs.env file, there is no config directory under $HADOOP_PREFIX/share/hadoop/yarn/webapps/ because condifurations default-config.js are build into ember deploy. There is no configurations to be passed externally since they are included in ember deploy package. So removing configs.env is intended to 1. Since configs.env is not used anymore, it can be removed. 2. By removing config directory in ember deployed package, we found there is no configurations to be passed from external. Does it make sense? But if any other new configurations are introduced which should be changed from external, we need to implement a way to pass some values to deployed ember package. Do you consider of such future usage?
        Hide
        sunilg Sunil G added a comment -

        As per below doc, under hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md

        *In $HADOOP_PREFIX/share/hadoop/yarn/webapps/rm/config/configs.env*
        	- Update timelineWebAddress and rmWebAddress to the actual addresses run resource manager and timeline server
        	- If you run RM locally in you computer just for test purpose, you need to keep `corsproxy` running. Otherwise, you need to set `localBaseAddress` to empty.
        

        This is the help or readme doc which explains how to configure YARN UI in a real production cluster. WE will not be editing any .js files to change config. It should be done via configs.env itself. You can also refer TEZ project for same.

        I could say that defaul-config.js is no longer needed anymore. I can try remove the same in another ticket.

        Show
        sunilg Sunil G added a comment - As per below doc, under hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md *In $HADOOP_PREFIX/share/hadoop/yarn/webapps/rm/config/configs.env* - Update timelineWebAddress and rmWebAddress to the actual addresses run resource manager and timeline server - If you run RM locally in you computer just for test purpose, you need to keep `corsproxy` running. Otherwise, you need to set `localBaseAddress` to empty. This is the help or readme doc which explains how to configure YARN UI in a real production cluster. WE will not be editing any .js files to change config. It should be done via configs.env itself. You can also refer TEZ project for same. I could say that defaul-config.js is no longer needed anymore. I can try remove the same in another ticket.
        Hide
        lewuathe Kai Sasaki added a comment -

        Sunil G I see. Sorry for my misunderstanding and missed the documentation. I'll update to use configs.env under HADOOP_CONF_DIR as described initially. Thanks you so much for clear explanation!

        Show
        lewuathe Kai Sasaki added a comment - Sunil G I see. Sorry for my misunderstanding and missed the documentation. I'll update to use configs.env under HADOOP_CONF_DIR as described initially. Thanks you so much for clear explanation!
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Kai Sasaki and Sunil G looking into the issue.

        I'm not sure about how to solve this issue:

        To me, It is fine to put configs.env in any place for a development environment. For example under source tree.

        But it is still very important to make it be able to config envs from HADOOP_CONF_DIR. And I think it's better to not assume source code dir: $HADOOP_PREFIX/share/hadoop/yarn/webapps/ will be writable in a real environment. What I can mostly see for most deployment is, yarn user doesn't have permission to write $HADOOP_PREFIX.

        Can we pass some parameter/env to the HTTP server which hosts YARN UI code in RM?

        Sreenath Somarajapuram, could you also share some thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Kai Sasaki and Sunil G looking into the issue. I'm not sure about how to solve this issue: To me, It is fine to put configs.env in any place for a development environment. For example under source tree. But it is still very important to make it be able to config envs from HADOOP_CONF_DIR. And I think it's better to not assume source code dir: $HADOOP_PREFIX/share/hadoop/yarn/webapps/ will be writable in a real environment. What I can mostly see for most deployment is, yarn user doesn't have permission to write $HADOOP_PREFIX. Can we pass some parameter/env to the HTTP server which hosts YARN UI code in RM? Sreenath Somarajapuram , could you also share some thoughts?
        Hide
        cheersyang Weiwei Yang added a comment -

        Hello Wangda Tan

        If all configuration are set in yarn-site.xml, it is possible to retrieve them via http://RM:8088/conf. I am currently working on a JIRA to improve the way of getting configuration property via REST call, this would help this case. See more from HADOOP-13628. For example, load a property to UI code would simply call http://RM:8088/conf?name=yarn.property.name, it will return a json format response (when accept header is set to json).

        Show
        cheersyang Weiwei Yang added a comment - Hello Wangda Tan If all configuration are set in yarn-site.xml, it is possible to retrieve them via http://RM:8088/conf . I am currently working on a JIRA to improve the way of getting configuration property via REST call, this would help this case. See more from HADOOP-13628 . For example, load a property to UI code would simply call http://RM:8088/conf?name=yarn.property.name , it will return a json format response (when accept header is set to json).
        Hide
        Sreenath Sreenath Somarajapuram added a comment -

        Wangda Tan

        config.env, default-config.js and environment.js were initially created with different purposes.

        1. config.env - For configuring host URLs and other values respective to the machine the UI is hosted in. It would be a separate file, and wont be imploded by the build script. Hence when the UI is hosted in tomcat or some similar server, the user can go and configure the UI. In other words, this file is used to configure UI in the deployment phase.
        2. default-config.js - Contains default values of the above + other constants internal to the UI (Like REST end point namespaces). This would be imploded and minified by the build script. Hence this file can be changed only before the build, or in the development phase.
        3. environment.js - This is Ember's standard file, and contain Ember related environment variables. They define how ember works.

        In short. If we remove config.env. Someone who want to host our UI war in his/her tomcat or similar servers would find it hard to get the UI working.

        Now accessing the configuration:

        1. Production - UI hosted in RM : As mentioned by Weiwei Yang, the UI can load configurations from RM's conf endpoint.
        2. Production - UI hosted in a web server (tomcat) : config.env can be used to configure the host URLs. (HADOOP_CONF_DIR could be out of reach as the machine its hosted in need not be even a part of the cluster)
        3. Production - wrapped : Things would be different if it is viewed inside other web interfaces like Ambari. We would have to take the configurations from REST end points provided by the respective interfaces. Guess that is out of scope now.
        4. Development : config.env can be used to configure the host URLs.
        Show
        Sreenath Sreenath Somarajapuram added a comment - Wangda Tan config.env, default-config.js and environment.js were initially created with different purposes. config.env - For configuring host URLs and other values respective to the machine the UI is hosted in. It would be a separate file, and wont be imploded by the build script. Hence when the UI is hosted in tomcat or some similar server, the user can go and configure the UI. In other words, this file is used to configure UI in the deployment phase. default-config.js - Contains default values of the above + other constants internal to the UI (Like REST end point namespaces). This would be imploded and minified by the build script. Hence this file can be changed only before the build, or in the development phase. environment.js - This is Ember's standard file, and contain Ember related environment variables. They define how ember works. In short. If we remove config.env. Someone who want to host our UI war in his/her tomcat or similar servers would find it hard to get the UI working. Now accessing the configuration: Production - UI hosted in RM : As mentioned by Weiwei Yang , the UI can load configurations from RM's conf endpoint. Production - UI hosted in a web server (tomcat) : config.env can be used to configure the host URLs. (HADOOP_CONF_DIR could be out of reach as the machine its hosted in need not be even a part of the cluster) Production - wrapped : Things would be different if it is viewed inside other web interfaces like Ambari. We would have to take the configurations from REST end points provided by the respective interfaces. Guess that is out of scope now. Development : config.env can be used to configure the host URLs.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Weiwei Yang, for the REST API enhancement, it will be very helpful for retrieving configuration values!

        Sreenath Somarajapuram thanks for your suggestions, I'm still trying to understand how it works. For now, I think the most important use case for the new UI is:

        1. Production - UI hosted in RM

        I can understand the UI can get configs from REST API. But how the UI knows which is the address to RM REST endpoint? If user update the default web port from 8088 to other port, or on a multi-homing environment, RM hosts on a different hostname, how the UI talks to RM?
        Is there any approach in JS/ember that we can pass some environment variables while hosting the UI code?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Weiwei Yang , for the REST API enhancement, it will be very helpful for retrieving configuration values! Sreenath Somarajapuram thanks for your suggestions, I'm still trying to understand how it works. For now, I think the most important use case for the new UI is: 1. Production - UI hosted in RM I can understand the UI can get configs from REST API. But how the UI knows which is the address to RM REST endpoint? If user update the default web port from 8088 to other port, or on a multi-homing environment, RM hosts on a different hostname, how the UI talks to RM? Is there any approach in JS/ember that we can pass some environment variables while hosting the UI code?
        Hide
        Sreenath Sreenath Somarajapuram added a comment -

        Wangda Tan
        Like the existing UI that is available at http://RM:8088/cluster endpoint, wouldn't the new UI be hosted under 8088 itself? In JS the domain an UI is loaded from is available @ window.location.host.
        In other words, if the UI is loaded from http://<host-name>:<port>/ui, the UI can get the required details from window.location.host (which would be http://<host-name>:<port>), and connect to http://<host-name>:<port>/conf.

        Show
        Sreenath Sreenath Somarajapuram added a comment - Wangda Tan Like the existing UI that is available at http://RM:8088/cluster endpoint, wouldn't the new UI be hosted under 8088 itself? In JS the domain an UI is loaded from is available @ window.location.host. In other words, if the UI is loaded from http://<host-name>:<port>/ui, the UI can get the required details from window.location.host (which would be http://<host-name>:<port>), and connect to http://<host-name>:<port>/conf.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sreenath Somarajapuram,

        Currently we load the new UI to the port RM:8288, since we want to keep both of new UI and old UI, and new UI can be enabled by user manually.

        I'm also OK (and actually preferred) to run the new UI on the same port of the RM, such as RM:8088/ui, since we're using a different Jetty server to host the new UI, not sure if it is possible to run two HTTP server on the same port but has different namespace. Did you have experiences about that? If you haven't tried it before, I can also take a look.

        Show
        leftnoteasy Wangda Tan added a comment - Sreenath Somarajapuram , Currently we load the new UI to the port RM:8288, since we want to keep both of new UI and old UI, and new UI can be enabled by user manually. I'm also OK (and actually preferred) to run the new UI on the same port of the RM, such as RM:8088/ui, since we're using a different Jetty server to host the new UI, not sure if it is possible to run two HTTP server on the same port but has different namespace. Did you have experiences about that? If you haven't tried it before, I can also take a look.
        Hide
        Sreenath Sreenath Somarajapuram added a comment -

        not sure if it is possible to run two HTTP server on the same port

        Not sure why we need two servers. Old UI had various static assets like the hadoop-st.png logo file, yarn.css and various js files. Similar to them all our new UI files are static assets, and we just need to server them over '/ui' end point when enabled by the user.

        Show
        Sreenath Sreenath Somarajapuram added a comment - not sure if it is possible to run two HTTP server on the same port Not sure why we need two servers. Old UI had various static assets like the hadoop-st.png logo file, yarn.css and various js files. Similar to them all our new UI files are static assets, and we just need to server them over '/ui' end point when enabled by the user.
        Hide
        Sreenath Sreenath Somarajapuram added a comment -

        Wangda Tan Weiwei Yang Adding to that, would be great if /conf support content-type, JSON. Have created YARN-5668 for the same.

        Show
        Sreenath Sreenath Somarajapuram added a comment - Wangda Tan Weiwei Yang Adding to that, would be great if /conf support content-type, JSON. Have created YARN-5668 for the same.
        Hide
        sunilg Sunil G added a comment -

        Hi All

        Few thoughts from my side.

        • I think overall approach is to get the configuration (timeline or others etc) from new REST api (RM side). So to get initial RM address, could we pass it as some environment variable?
        • To run both old and new ui under same port, we do not need 2 HTTP servers. I think new namespace could be used for that. But I am not very sure about complexities. Its better we can some how run new UI on another port, and get the initial RM address via some common means (env variable, file etc).
        Show
        sunilg Sunil G added a comment - Hi All Few thoughts from my side. I think overall approach is to get the configuration (timeline or others etc) from new REST api (RM side). So to get initial RM address, could we pass it as some environment variable? To run both old and new ui under same port, we do not need 2 HTTP servers. I think new namespace could be used for that. But I am not very sure about complexities. Its better we can some how run new UI on another port, and get the initial RM address via some common means (env variable, file etc).
        Hide
        Sreenath Sreenath Somarajapuram added a comment -

        configuration (timeline or others etc) from new REST api

        Already /conf is working, guess we just need to use that.

        Its better we can some how run new UI on another port

        In a case the UI must be server from a different port, say http://RM:8089:/cluster, just provide a http://RM:8089/conf (An alias if http://RM:8088/conf) and things would be well and good.

        Show
        Sreenath Sreenath Somarajapuram added a comment - configuration (timeline or others etc) from new REST api Already /conf is working, guess we just need to use that. Its better we can some how run new UI on another port In a case the UI must be server from a different port, say http://RM:8089:/cluster , just provide a http://RM:8089/conf (An alias if http://RM:8088/conf ) and things would be well and good.
        Hide
        lewuathe Kai Sasaki added a comment -

        Hi Sunil G.

        I think overall approach is to get the configuration (timeline or others etc) from new REST api (RM side). So to get initial RM address, could we pass it as some environment variable?

        It sounds reasonable for me. The configurations except for REST API host:port can be obtained and REST API host:port is configured by environment variable. This way does not add so much complexity and we can delegate the configuration management responsibility to REST API not Yarn UI v2 side.

        Show
        lewuathe Kai Sasaki added a comment - Hi Sunil G . I think overall approach is to get the configuration (timeline or others etc) from new REST api (RM side). So to get initial RM address, could we pass it as some environment variable? It sounds reasonable for me. The configurations except for REST API host:port can be obtained and REST API host:port is configured by environment variable. This way does not add so much complexity and we can delegate the configuration management responsibility to REST API not Yarn UI v2 side.
        Hide
        cheersyang Weiwei Yang added a comment -

        Hello Sreenath Somarajapuram

        I think this is already done in HADOOP-13588, could you please take a look?

        Show
        cheersyang Weiwei Yang added a comment - Hello Sreenath Somarajapuram I think this is already done in HADOOP-13588 , could you please take a look?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks for sharing your thoughts, Sreenath Somarajapuram, Sunil G, Weiwei Yang, Kai Sasaki.

        I think fetch configs from REST server is a great solution, we can completely avoid adding new config files to hadoop conf dir. Now we have two choice:

        • Use existing port, and run UI server under a separate namespace
        • Use a new port, and host conf/ on the new server.

        Both solutions look fine to me, but I think the first solution might be better since it avoid adding a new config and all existing security features should (need verify) be kept. I will spend some time today to see if it is possible to host UI under existing port, will keep this thread posted.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks for sharing your thoughts, Sreenath Somarajapuram , Sunil G , Weiwei Yang , Kai Sasaki . I think fetch configs from REST server is a great solution, we can completely avoid adding new config files to hadoop conf dir. Now we have two choice: Use existing port, and run UI server under a separate namespace Use a new port, and host conf/ on the new server. Both solutions look fine to me, but I think the first solution might be better since it avoid adding a new config and all existing security features should (need verify) be kept. I will spend some time today to see if it is possible to host UI under existing port, will keep this thread posted.
        Hide
        sunilg Sunil G added a comment -

        Currently /logs has added as a new context to existing webserver in RM. Similarly we could try adding a new Context for new web UI. I am making some prototype for same, and will update status here.

        Show
        sunilg Sunil G added a comment - Currently /logs has added as a new context to existing webserver in RM. Similarly we could try adding a new Context for new web UI. I am making some prototype for same, and will update status here.
        Hide
        sunilg Sunil G added a comment -

        Attaching a POC patch to host new web UI under same old webapp port.

        http://<RMWebAddress:port>/newUI/
        

        Also attaching a screen shot.. I think we can choose this model..

        Thoughts?

        Show
        sunilg Sunil G added a comment - Attaching a POC patch to host new web UI under same old webapp port. http://<RMWebAddress:port>/newUI/ Also attaching a screen shot.. I think we can choose this model.. Thoughts?
        Hide
        sunilg Sunil G added a comment -

        I think hosting new web UI under same old webapp port can be tracked via another ticket. I ll raise one ticket if there are no objections.

        Show
        sunilg Sunil G added a comment - I think hosting new web UI under same old webapp port can be tracked via another ticket. I ll raise one ticket if there are no objections.
        Hide
        sunilg Sunil G added a comment -

        With YARN-5698 we can host new ui under same old webapp port. Kai Sasaki, will you be able to fetch new config from RM on top of that patch. If you dont have bandwidth, i could help.

        Show
        sunilg Sunil G added a comment - With YARN-5698 we can host new ui under same old webapp port. Kai Sasaki , will you be able to fetch new config from RM on top of that patch. If you dont have bandwidth, i could help.
        Hide
        sunilg Sunil G added a comment -

        Attaching an initial patch.

        This will download RM address and Timeline address as per HADOOP-13628.

        Some more tests are needed .I ll update progress here. Kai Sasaki, if you are not planning on this, I ll try taking over this.

        Wangda Tan and Sreenath Somarajapuram. pls help to check the approach.

        Show
        sunilg Sunil G added a comment - Attaching an initial patch. This will download RM address and Timeline address as per HADOOP-13628 . Some more tests are needed .I ll update progress here. Kai Sasaki , if you are not planning on this, I ll try taking over this. Wangda Tan and Sreenath Somarajapuram . pls help to check the approach.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 4m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 0m 0s The patch 6 line(s) with tabs.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        4m 57s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b666617
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832960/YARN-5145-YARN-3368.02.patch
        JIRA Issue YARN-5145
        Optional Tests asflicense
        uname Linux 7b56f517a87a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-3368 / 1e47518
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13363/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13363/artifact/patchprocess/whitespace-tabs.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13363/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 4m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 0s The patch 6 line(s) with tabs. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 4m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:b666617 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832960/YARN-5145-YARN-3368.02.patch JIRA Issue YARN-5145 Optional Tests asflicense uname Linux 7b56f517a87a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-3368 / 1e47518 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13363/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13363/artifact/patchprocess/whitespace-tabs.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui Console output https://builds.apache.org/job/PreCommit-YARN-Build/13363/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Took a quick try for patch, on my local single node env, it reports

        XMLHttpRequest cannot load http://localhost:8188/ws/v1/applicationhistory/apps/application_1476304805106_0004/appattempts/appattempt_1476304805106_0004_000001/containers. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8088' is therefore not allowed access.

        It doesn't work after I run the {{corsproxy} as well. Do we have CRFS config for timeline server? If no, is there any solution that we can run yarn ui v2 with timeline server enabled on the same node?

        Show
        leftnoteasy Wangda Tan added a comment - Took a quick try for patch, on my local single node env, it reports XMLHttpRequest cannot load http://localhost:8188/ws/v1/applicationhistory/apps/application_1476304805106_0004/appattempts/appattempt_1476304805106_0004_000001/containers . No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8088' is therefore not allowed access. It doesn't work after I run the {{corsproxy} as well. Do we have CRFS config for timeline server? If no, is there any solution that we can run yarn ui v2 with timeline server enabled on the same node?
        Hide
        sunilg Sunil G added a comment - - edited

        Thanks Wangda Tan for sharing the test results..

        I think issue was with CORS from client side. Client also has to query with cors address.

        I have made necessary changes and tested in 5node cluster where timeline v2 server is running. RM has some config to avoid cors issue, however timeline v2 has cors issue hence we must use corsproxy for timeline v2 for a brief time till timeline supports cors.

        Hence I have made cors as mandatory in this patch. Pls help to check the same.. cc/Wangda Tan and Sreenath Somarajapuram

        Show
        sunilg Sunil G added a comment - - edited Thanks Wangda Tan for sharing the test results.. I think issue was with CORS from client side. Client also has to query with cors address. I have made necessary changes and tested in 5node cluster where timeline v2 server is running. RM has some config to avoid cors issue, however timeline v2 has cors issue hence we must use corsproxy for timeline v2 for a brief time till timeline supports cors. Hence I have made cors as mandatory in this patch. Pls help to check the same.. cc/ Wangda Tan and Sreenath Somarajapuram
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 3m 43s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 0m 0s The patch 51 line(s) with tabs.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        4m 16s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b666617
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833137/YARN-5145-YARN-3368.03.patch
        JIRA Issue YARN-5145
        Optional Tests asflicense
        uname Linux 3b401ae43533 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-3368 / 60c8810
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13378/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13378/artifact/patchprocess/whitespace-tabs.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13378/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 3m 43s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 0s The patch 51 line(s) with tabs. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 4m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:b666617 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833137/YARN-5145-YARN-3368.03.patch JIRA Issue YARN-5145 Optional Tests asflicense uname Linux 3b401ae43533 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-3368 / 60c8810 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13378/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13378/artifact/patchprocess/whitespace-tabs.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui Console output https://builds.apache.org/job/PreCommit-YARN-Build/13378/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Offline discussed with Sunil G,

        The CORS is supported by timeline server v1 only, since now the ats v2 integration to web ui is not merged to this branch, I think we don't need hardcoding proxy settings in the code.

        Instead, we should add support of CORS to ats v2 so we don't need do hard coding in ui code after ats v2 code merged.

        And format of the file seems not correct, it mixed tabs and white spaces.

        Show
        leftnoteasy Wangda Tan added a comment - Offline discussed with Sunil G , The CORS is supported by timeline server v1 only, since now the ats v2 integration to web ui is not merged to this branch, I think we don't need hardcoding proxy settings in the code. Instead, we should add support of CORS to ats v2 so we don't need do hard coding in ui code after ats v2 code merged. And format of the file seems not correct, it mixed tabs and white spaces.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan. Updating patch after addressing the comments.

        cc/Sreenath Somarajapuram

        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan . Updating patch after addressing the comments. cc/ Sreenath Somarajapuram
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 4m 30s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        5m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b666617
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833395/YARN-5145-YARN-3368.04.patch
        JIRA Issue YARN-5145
        Optional Tests asflicense
        uname Linux dfd3f9d21dfa 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-3368 / 164a3c2
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13395/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 4m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 5m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:b666617 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833395/YARN-5145-YARN-3368.04.patch JIRA Issue YARN-5145 Optional Tests asflicense uname Linux dfd3f9d21dfa 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-3368 / 164a3c2 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui Console output https://builds.apache.org/job/PreCommit-YARN-Build/13395/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Sreenath Sreenath Somarajapuram added a comment - - edited
        • Minor comment, Ember Logger could be used instead of direct console log.
        • Looking at
          if(address == "0.0.0.0") {

          , just wondering if we could have condition where the addres comes as localhost or 127.0.0.1

        Show
        Sreenath Sreenath Somarajapuram added a comment - - edited Minor comment, Ember Logger could be used instead of direct console log. Looking at if (address == "0.0.0.0" ) { , just wondering if we could have condition where the addres comes as localhost or 127.0.0.1
        Hide
        sunilg Sunil G added a comment -

        Thanks Sreenath Somarajapuram

        Addressed the comments.. However I think we can compare only localhost/0.0.0.0 for now.. Handling 127...* family of address may not be veery clean. Thoughts /cc Wangda Tan

        Show
        sunilg Sunil G added a comment - Thanks Sreenath Somarajapuram Addressed the comments.. However I think we can compare only localhost/0.0.0.0 for now.. Handling 127. . .* family of address may not be veery clean. Thoughts /cc Wangda Tan
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        0m 56s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b666617
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833488/YARN-5145-YARN-3368.05.patch
        JIRA Issue YARN-5145
        Optional Tests asflicense
        uname Linux c5e0c664e5cb 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-3368 / 164a3c2
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13402/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 0m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:b666617 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833488/YARN-5145-YARN-3368.05.patch JIRA Issue YARN-5145 Optional Tests asflicense uname Linux c5e0c664e5cb 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-3368 / 164a3c2 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui Console output https://builds.apache.org/job/PreCommit-YARN-Build/13402/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Attaching new patch to handle timeline address configuration from configs.env

        Show
        sunilg Sunil G added a comment - Attaching new patch to handle timeline address configuration from configs.env
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        1m 1s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b666617
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833721/YARN-5145-YARN-3368.06.patch
        JIRA Issue YARN-5145
        Optional Tests asflicense
        uname Linux a87496b09e01 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-3368 / 164a3c2
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13406/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 1m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:b666617 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833721/YARN-5145-YARN-3368.06.patch JIRA Issue YARN-5145 Optional Tests asflicense uname Linux a87496b09e01 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-3368 / 164a3c2 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui Console output https://builds.apache.org/job/PreCommit-YARN-Build/13406/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Commit to YARN-3368 branch, thanks Kai Sasaki and Sunil G for the patch and thanks Sreenath Somarajapuram for review!

        Show
        leftnoteasy Wangda Tan added a comment - Commit to YARN-3368 branch, thanks Kai Sasaki and Sunil G for the patch and thanks Sreenath Somarajapuram for review!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10778 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10778/)
        YARN-5145. YARN-3368 Move new YARN UI configuration to (wangda: rev d8be7667597b468865e3d888cb3663c1d78823ec)

        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/tests/unit/initializers/loader-test.js
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/initializers/loader.js
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10778 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10778/ ) YARN-5145 . YARN-3368 Move new YARN UI configuration to (wangda: rev d8be7667597b468865e3d888cb3663c1d78823ec) (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/tests/unit/initializers/loader-test.js (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/initializers/loader.js

          People

          • Assignee:
            sunilg Sunil G
            Reporter:
            leftnoteasy Wangda Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development