Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3294

Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: capacityscheduler
    • Labels:
      None
    • Target Version/s:

      Description

      It would be nice to have a button on the web UI that would allow dumping of debug logs for just the capacity scheduler for a fixed period of time(1 min, 5 min or so) in a separate log file. It would be useful when debugging scheduler behavior without affecting the rest of the resourcemanager.

      1. apache-yarn-3294.0.patch
        17 kB
        Varun Vasudev
      2. apache-yarn-3294.1.patch
        17 kB
        Varun Vasudev
      3. apache-yarn-3294.2.patch
        14 kB
        Varun Vasudev
      4. apache-yarn-3294.3.patch
        14 kB
        Varun Vasudev
      5. apache-yarn-3294.4.patch
        14 kB
        Varun Vasudev
      6. Screen Shot 2015-03-12 at 8.51.25 PM.png
        377 kB
        Varun Vasudev

        Issue Links

          Activity

          Hide
          vvasudev Varun Vasudev added a comment -

          Thomas Graves - thanks for pointing out the admin issue. My apologies for missing it. I've filed YARN-3517 and updated it with a patch, which allows only admins to use the functionality. Can you please review and leave comments there?

          Show
          vvasudev Varun Vasudev added a comment - Thomas Graves - thanks for pointing out the admin issue. My apologies for missing it. I've filed YARN-3517 and updated it with a patch, which allows only admins to use the functionality. Can you please review and leave comments there?
          Hide
          tgraves Thomas Graves added a comment -

          Xuan Gong Varun Vasudev I saw this show up in the UI on branch-2. I don't see any permissions checks on this, am I perhaps missing it? We don't want arbitrary users to be able to change log level on the RM. They could slow it down and cause disks to fill up.

          I also don't see an option to disable this, is there one? If not I think we want it.

          Honestly I don't really see a need for this button at all as you can change in the logLevel app. But since its in we atleast need to protect it and in my opinion disable it for normal users.

          Show
          tgraves Thomas Graves added a comment - Xuan Gong Varun Vasudev I saw this show up in the UI on branch-2. I don't see any permissions checks on this, am I perhaps missing it? We don't want arbitrary users to be able to change log level on the RM. They could slow it down and cause disks to fill up. I also don't see an option to disable this, is there one? If not I think we want it. Honestly I don't really see a need for this button at all as you can change in the logLevel app. But since its in we atleast need to protect it and in my opinion disable it for normal users.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/)
          YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/ ) YARN-3294 . Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/)
          YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/ ) YARN-3294 . Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/)
          YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/ ) YARN-3294 . Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/)
          YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/ ) YARN-3294 . Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/)
          YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/ ) YARN-3294 . Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/)
          YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/ ) YARN-3294 . Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7521 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7521/)
          YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7521 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7521/ ) YARN-3294 . Allow dumping of Capacity Scheduler debug logs via web UI for (xgong: rev d27e9241e8676a0edb2d35453cac5f9495fcd605) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestAdHocLogDumper.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AdHocLogDumper.java
          Hide
          xgong Xuan Gong added a comment -

          Committed into trunk/branch-2. Thanks, varun.

          Show
          xgong Xuan Gong added a comment - Committed into trunk/branch-2. Thanks, varun.
          Hide
          xgong Xuan Gong added a comment -

          +1 lgtm. Will commit

          Show
          xgong Xuan Gong added a comment - +1 lgtm. Will commit
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707223/apache-yarn-3294.4.patch
          against trunk revision c906a1d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7102//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7102//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707223/apache-yarn-3294.4.patch against trunk revision c906a1d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7102//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7102//console This message is automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch to address the feedback by Jian He.

          how about passing the ‘logHierarchy’ as a parameter too ? so that the web service also work for user-provided class name, may be also take in logLevel and dumpLocation as a parameter too ?

          I would prefer not to have those as params. One of the concerns was that users will dump log files with different filenames which won't get rotated and use up disk space. Similarly with the hierarchy - we just end up with a slightly different version of the applet we already have.

          question - wondering whether we need the “@QueryParam” for the 'time' parameter, as other some other places in the same file do, e.g. getApps;

          Good point. I've marked it as a FormParam.

          if no permission for RM to write logs, will the webService throw error ?

          Yes, I've updated the Javascript code to throw up an alert letting you know there was an error with the request.

          new File(System.getProperty("hadoop.log.dir"), targetFilename);, should this be “yarn.log.dir” ?

          Fixed.

          return "Capacity scheduler logs are being created."; it should probably return JAXB formatted response. The response seems broken if I directly access the web service from my browser

          The method is a POST method, with no GET method - that's probably why you got the error. From my machine using curl,

          curl -i -X POST -d "time=60" http://localhost:8088/ws/v1/cluster/scheduler/logs
          HTTP/1.1 200 OK
          Cache-Control: no-cache
          Expires: Wed, 25 Mar 2015 02:07:33 GMT
          Date: Wed, 25 Mar 2015 02:07:33 GMT
          Pragma: no-cache
          Expires: Wed, 25 Mar 2015 02:07:33 GMT
          Date: Wed, 25 Mar 2015 02:07:33 GMT
          Pragma: no-cache
          Content-Type: application/json
          Transfer-Encoding: chunked
          Server: Jetty(6.1.26)
          
          Capacity scheduler logs are being created.
          

          AdHocLogDumper#appenderLevels - entry is never removed from the map, is this excepted?

          I was creating a new appenderLevels hash every time the dumpLogs function is called, changed to a clear now.

          appenderLevels.clear();
          

          may be mark private/unstable for the AdHocLogDumper for now

          Fixed.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch to address the feedback by Jian He . how about passing the ‘logHierarchy’ as a parameter too ? so that the web service also work for user-provided class name, may be also take in logLevel and dumpLocation as a parameter too ? I would prefer not to have those as params. One of the concerns was that users will dump log files with different filenames which won't get rotated and use up disk space. Similarly with the hierarchy - we just end up with a slightly different version of the applet we already have. question - wondering whether we need the “@QueryParam” for the 'time' parameter, as other some other places in the same file do, e.g. getApps; Good point. I've marked it as a FormParam. if no permission for RM to write logs, will the webService throw error ? Yes, I've updated the Javascript code to throw up an alert letting you know there was an error with the request. new File(System.getProperty("hadoop.log.dir"), targetFilename);, should this be “yarn.log.dir” ? Fixed. return "Capacity scheduler logs are being created."; it should probably return JAXB formatted response. The response seems broken if I directly access the web service from my browser The method is a POST method, with no GET method - that's probably why you got the error. From my machine using curl, curl -i -X POST -d "time=60" http://localhost:8088/ws/v1/cluster/scheduler/logs HTTP/1.1 200 OK Cache-Control: no-cache Expires: Wed, 25 Mar 2015 02:07:33 GMT Date: Wed, 25 Mar 2015 02:07:33 GMT Pragma: no-cache Expires: Wed, 25 Mar 2015 02:07:33 GMT Date: Wed, 25 Mar 2015 02:07:33 GMT Pragma: no-cache Content-Type: application/json Transfer-Encoding: chunked Server: Jetty(6.1.26) Capacity scheduler logs are being created. AdHocLogDumper#appenderLevels - entry is never removed from the map, is this excepted? I was creating a new appenderLevels hash every time the dumpLogs function is called, changed to a clear now. appenderLevels.clear(); may be mark private/unstable for the AdHocLogDumper for now Fixed.
          Hide
          jianhe Jian He added a comment - - edited

          Thanks for updating the patch, looks good overall,

          • how about passing the ‘logHierarchy’ as a parameter too ? so that the web service also work for user-provided class name, may be also take in logLevel and dumpLocation as a parameter too ?
               @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })

                public String dumpSchedulerLogs(String time) throws IOException {
            
          • question - wondering whether we need the “@QueryParam” for the 'time' parameter, as other some other places in the same file do, e.g. getApps;
          • if no permission for RM to write logs, will the webService throw error ?
          • new File(System.getProperty("hadoop.log.dir"), targetFilename);, should this be “yarn.log.dir” ?
          • return "Capacity scheduler logs are being created."; it should probably return JAXB formatted response. The response seems broken if I directly access the web service from my browser
          • AdHocLogDumper#appenderLevels - entry is never removed from the map, is this excepted?
          • may be mark private/unstable for the AdHocLogDumper for now
          Show
          jianhe Jian He added a comment - - edited Thanks for updating the patch, looks good overall, how about passing the ‘logHierarchy’ as a parameter too ? so that the web service also work for user-provided class name, may be also take in logLevel and dumpLocation as a parameter too ? @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
 public String dumpSchedulerLogs( String time) throws IOException { question - wondering whether we need the “@QueryParam” for the 'time' parameter, as other some other places in the same file do, e.g. getApps; if no permission for RM to write logs, will the webService throw error ? new File(System.getProperty("hadoop.log.dir"), targetFilename); , should this be “yarn.log.dir” ? return "Capacity scheduler logs are being created."; it should probably return JAXB formatted response. The response seems broken if I directly access the web service from my browser AdHocLogDumper#appenderLevels - entry is never removed from the map, is this excepted? may be mark private/unstable for the AdHocLogDumper for now
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705966/apache-yarn-3294.3.patch
          against trunk revision 1561231.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7044//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7044//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705966/apache-yarn-3294.3.patch against trunk revision 1561231. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7044//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7044//console This message is automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch to address findbug errors.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch to address findbug errors.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705928/apache-yarn-3294.2.patch
          against trunk revision 8041267.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestRM
          org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7042//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7042//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7042//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7042//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705928/apache-yarn-3294.2.patch against trunk revision 8041267. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRM org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7042//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7042//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7042//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7042//console This message is automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the review Jian!

          1. will it dump the logs for package “org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity” or “org.apache.hadoop.yarn.server.resourcemanager.scheduler”. I think it should be dumping latter as some scheduler logs sit in common

          Log log = LogFactory.getLog(CapacityScheduler.class.getPackage().getName());


          2. the newly added code can be in a separate class, maybe call it SchedulerLogDumper

          3. Overall, do you think it's useful to make this as a generic log dumper ? i.e. parameterize the class name of the log to be dumped.

          Addressed all three. I made it a standalone class called AdHocLogDumper that takes the log name and dumps the log. It now dumps the log for "org.apache.hadoop.yarn.server.resourcemanager.scheduler".

          private Logger packageLogger; does not need to be a global value

          Fixed.

          dumping logs is useful for large amount of logs. Do you think it's also useful to surface the logs on the web page.

          I think we should skip it for now. On a single node cluster with one pi job running, it dumped about 200K of logs in 1 min. I'm worried with larger clusters, the log size will be too large. The log will still be accessible via the /logs link.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the review Jian! 1. will it dump the logs for package “org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity” or “org.apache.hadoop.yarn.server.resourcemanager.scheduler”. I think it should be dumping latter as some scheduler logs sit in common Log log = LogFactory.getLog(CapacityScheduler.class.getPackage().getName());
 2. the newly added code can be in a separate class, maybe call it SchedulerLogDumper 3. Overall, do you think it's useful to make this as a generic log dumper ? i.e. parameterize the class name of the log to be dumped. Addressed all three. I made it a standalone class called AdHocLogDumper that takes the log name and dumps the log. It now dumps the log for "org.apache.hadoop.yarn.server.resourcemanager.scheduler". private Logger packageLogger; does not need to be a global value Fixed. dumping logs is useful for large amount of logs. Do you think it's also useful to surface the logs on the web page. I think we should skip it for now. On a single node cluster with one pi job running, it dumped about 200K of logs in 1 min. I'm worried with larger clusters, the log size will be too large. The log will still be accessible via the /logs link.
          Hide
          jianhe Jian He added a comment -

          thanks Varun, looks good overall.

          • will it dump the logs for package “org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity” or “org.apache.hadoop.yarn.server.resourcemanager.scheduler”. I think it should be dumping latter as some scheduler logs sit in common
            Log log = LogFactory.getLog(CapacityScheduler.class.getPackage().getName());

            
          • the newly added code can be in a separate class, maybe call it SchedulerLogDumper
          • private Logger packageLogger; does not need to be a global value
          • dumping logs is useful for large amount of logs. Do you think it's also useful to surface the logs on the web page.
          • Overall, do you think it's useful to make this as a generic log dumper ? i.e. parameterize the class name of the log to be dumped.
          Show
          jianhe Jian He added a comment - thanks Varun, looks good overall. will it dump the logs for package “org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity” or “org.apache.hadoop.yarn.server.resourcemanager.scheduler”. I think it should be dumping latter as some scheduler logs sit in common Log log = LogFactory.getLog(CapacityScheduler.class.getPackage().getName());
 the newly added code can be in a separate class, maybe call it SchedulerLogDumper private Logger packageLogger; does not need to be a global value dumping logs is useful for large amount of logs. Do you think it's also useful to surface the logs on the web page. Overall, do you think it's useful to make this as a generic log dumper ? i.e. parameterize the class name of the log to be dumped.
          Hide
          vvasudev Varun Vasudev added a comment -

          The findbugs errors are unrelated to the patch. The test failures are also unrelated as per my analysis and I'm unsure of the Javac warnings since they seem to be from files I didn't modify. Jian He, can you help me out and take a look at the patch?

          Show
          vvasudev Varun Vasudev added a comment - The findbugs errors are unrelated to the patch. The test failures are also unrelated as per my analysis and I'm unsure of the Javac warnings since they seem to be from files I didn't modify. Jian He , can you help me out and take a look at the patch?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704374/apache-yarn-3294.1.patch
          against trunk revision 387f271.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1177 javac compiler warnings (more than the trunk's current 1152 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.ha.TestActiveStandbyElectorRealZK
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart

          The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.metrics2.lib.TestMutableMetrics
          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestTestTests
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6952//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6952//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6952//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6952//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704374/apache-yarn-3294.1.patch against trunk revision 387f271. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1177 javac compiler warnings (more than the trunk's current 1152 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.ha.TestActiveStandbyElectorRealZK org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.metrics2.lib.TestMutableMetrics org.apache.hadoop.yarn.server.resourcemanager.TestRMRestTestTests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6952//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6952//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6952//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6952//console This message is automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploading patch again to kick off jenkins.

          Show
          vvasudev Varun Vasudev added a comment - Uploading patch again to kick off jenkins.
          Hide
          vvasudev Varun Vasudev added a comment -

          The uploaded patch will dump debug logs for the capacity scheduler to "yarn-capacity-scheduler-debug.log" in the logs directory. Older logs will be overwritten and not rotated.

          Show
          vvasudev Varun Vasudev added a comment - The uploaded patch will dump debug logs for the capacity scheduler to "yarn-capacity-scheduler-debug.log" in the logs directory. Older logs will be overwritten and not rotated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded screenshot with the changes.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded screenshot with the changes.
          Hide
          vvasudev Varun Vasudev added a comment -

          I meant applet not servlet in the comment above.

          Show
          vvasudev Varun Vasudev added a comment - I meant applet not servlet in the comment above.
          Hide
          vvasudev Varun Vasudev added a comment -

          We've been seeing this repeating pattern when trying to debug customer clusters. We essentially do what you mention - go to the servlet, set the level to DEBUG for the scheduler for some time, set the log level back to INFO, then ask customers to send us the resourcemanager log, and grep/cut out the relevant sections. I was hoping that we could just tell them to click the button, dump the scheduler logs for this brief period of time to a different file and send us the file.

          Show
          vvasudev Varun Vasudev added a comment - We've been seeing this repeating pattern when trying to debug customer clusters. We essentially do what you mention - go to the servlet, set the level to DEBUG for the scheduler for some time, set the log level back to INFO, then ask customers to send us the resourcemanager log, and grep/cut out the relevant sections. I was hoping that we could just tell them to click the button, dump the scheduler logs for this brief period of time to a different file and send us the file.
          Hide
          jlowe Jason Lowe added a comment -

          Do we really need a dedicated button for a specific system/scheduler when there's already the logLevel applet that let's us control log levels of arbitrary loggers in the process?

          Show
          jlowe Jason Lowe added a comment - Do we really need a dedicated button for a specific system/scheduler when there's already the logLevel applet that let's us control log levels of arbitrary loggers in the process?

            People

            • Assignee:
              vvasudev Varun Vasudev
              Reporter:
              vvasudev Varun Vasudev
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development