Hadoop YARN
  1. Hadoop YARN
  2. YARN-151

Browser thinks RM main page JS is taking too long

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha, 0.23.3, 0.23.4
    • Fix Version/s: 3.0.0, 2.0.3-alpha, 0.23.5
    • Component/s: None
    • Labels:
      None

      Description

      The main RM page with the default settings of 10,000 applications can cause browsers to think that the JS on the page is stuck and ask you if you want to kill it. This is a big usability problem.

      1. YARN-151.patch
        8 kB
        Ravi Prakash
      2. testDeferred.tar.gz
        173 kB
        Ravi Prakash
      3. test.html
        4.02 MB
        Ravi Prakash

        Issue Links

          Activity

          Hide
          Luke Lu added a comment -

          The most straight forward fix would be upgrading to a more recent version of datatables and turn on defered rendering (not creating dom nodes for all rows, even they are hidden): http://datatables.net/release-datatables/examples/ajax/defer_render.html

          Sorting, searching through in js itself can easily handle up to 100k rows on modest hardware.

          Show
          Luke Lu added a comment - The most straight forward fix would be upgrading to a more recent version of datatables and turn on defered rendering (not creating dom nodes for all rows, even they are hidden): http://datatables.net/release-datatables/examples/ajax/defer_render.html Sorting, searching through in js itself can easily handle up to 100k rows on modest hardware.
          Hide
          Ravi Prakash added a comment -

          @Luke: Even if we used deferred rendering, it seems like we would still need to ship all that data to the browser, even if the user doesn't require it (e.g. the very first page of the JHS)

          Wouldn't server side be better? http://datatables.net/release-datatables/examples/data_sources/server_side.html It seems to be designed for exactly our use case.

          Show
          Ravi Prakash added a comment - @Luke: Even if we used deferred rendering, it seems like we would still need to ship all that data to the browser, even if the user doesn't require it (e.g. the very first page of the JHS) Wouldn't server side be better? http://datatables.net/release-datatables/examples/data_sources/server_side.html It seems to be designed for exactly our use case.
          Hide
          Luke Lu added a comment -

          10k apps is a small amount of data (a few hundred KB to a few MB) and that you don't need implement the logic (api that supports searching/sorting/paging) at server side and that you have better user response time (instant search experience) as the data is at the client side. The server side stuff would be better if we need to support more than 100K apps. If you come up with a patch, I'm happy to review it

          Show
          Luke Lu added a comment - 10k apps is a small amount of data (a few hundred KB to a few MB) and that you don't need implement the logic (api that supports searching/sorting/paging) at server side and that you have better user response time (instant search experience) as the data is at the client side. The server side stuff would be better if we need to support more than 100K apps. If you come up with a patch, I'm happy to review it
          Hide
          Ravi Prakash added a comment -

          Hi Luke! Thanks a lot for your prompt reply. 20000 jobs on the JHS resulted in a ~13 Mb file. (I can see that there's a lot of whitespace in the file, which we can probably get rid of). Anyway, I timed how much time it takes for the browser to download and render this on my machine. It took about 35 seconds. When I rendered the locally saved file, it took 11 seconds. Here's an HTML file containing a table with 20000 rows. It again took 12 seconds.

          It seems to me that the majority of the time (35-12) was taken in downloading the file. I don't know why it would take so long, but it did

          I'll try to do some more tests with DataTables with deferred rendering. If that is fast, I would like it too, so that I can graph that data in MAPREDUCE-4747. Killing two stones with 1 bird (since I don't like killing birds)

          Show
          Ravi Prakash added a comment - Hi Luke! Thanks a lot for your prompt reply. 20000 jobs on the JHS resulted in a ~13 Mb file. (I can see that there's a lot of whitespace in the file, which we can probably get rid of). Anyway, I timed how much time it takes for the browser to download and render this on my machine. It took about 35 seconds. When I rendered the locally saved file, it took 11 seconds. Here's an HTML file containing a table with 20000 rows. It again took 12 seconds. It seems to me that the majority of the time (35-12) was taken in downloading the file. I don't know why it would take so long, but it did I'll try to do some more tests with DataTables with deferred rendering. If that is fast, I would like it too, so that I can graph that data in MAPREDUCE-4747 . Killing two stones with 1 bird (since I don't like killing birds)
          Hide
          Ravi Prakash added a comment -

          Ok Luke! I'm convinced. Deferred rendering is AWESOME! Attaching the small test file(s) I made. 20,000 rows, rendered in 1-2 seconds. I even tried 200,000 rows. That took 5-6 seconds.

          Do we know how much we want the JHS to scale? Ideally ofcourse we'd like to keep all the jobs that have ever been run, and for that, we might want to use server_side, but deferred rendering seems fine for now.

          I'll work on creating a patch for using deferred rendering in the mean time. Thanks a ton!

          Show
          Ravi Prakash added a comment - Ok Luke! I'm convinced. Deferred rendering is AWESOME! Attaching the small test file(s) I made. 20,000 rows, rendered in 1-2 seconds. I even tried 200,000 rows. That took 5-6 seconds. Do we know how much we want the JHS to scale? Ideally ofcourse we'd like to keep all the jobs that have ever been run, and for that, we might want to use server_side, but deferred rendering seems fine for now. I'll work on creating a patch for using deferred rendering in the mean time. Thanks a ton!
          Hide
          Ravi Prakash added a comment -

          The same patch applies to all 3 branches : 0.23, branch-2, trunk

          Show
          Ravi Prakash added a comment - The same patch applies to all 3 branches : 0.23, branch-2, trunk
          Hide
          Ravi Prakash added a comment -

          Can someone please review and commit this?

          Show
          Ravi Prakash added a comment - Can someone please review and commit this?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553740/YARN-151.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/150//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/150//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553740/YARN-151.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/150//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/150//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          The patch looks good to me +1.

          I'll check it in.

          Show
          Robert Joseph Evans added a comment - The patch looks good to me +1. I'll check it in.
          Hide
          Robert Joseph Evans added a comment -

          Thank Ravi,

          I put this into trunk, branch-2, and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thank Ravi, I put this into trunk, branch-2, and branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3033 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3033/)
          YARN-151. Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410564)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410564
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3033 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3033/ ) YARN-151 . Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410564) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410564 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #39 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/39/)
          YARN-151. Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410564)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410564
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #39 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/39/ ) YARN-151 . Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410564) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410564 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #438 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/438/)
          svn merge -c 1410564 FIXES: YARN-151. Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410566)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410566
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #438 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/438/ ) svn merge -c 1410564 FIXES: YARN-151 . Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410566) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410566 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1229 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1229/)
          YARN-151. Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410564)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410564
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1229 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1229/ ) YARN-151 . Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410564) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410564 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1260 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1260/)
          YARN-151. Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410564)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410564
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1260 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1260/ ) YARN-151 . Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) (Revision 1410564) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410564 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java

            People

            • Assignee:
              Ravi Prakash
              Reporter:
              Robert Joseph Evans
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development