Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2934

Improve handling of container's stderr

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Most YARN applications redirect stderr to some file. That's why when container launch fails with ExitCodeException the message is empty.

      1. YARN-2934.v1.001.patch
        8 kB
        Naganarasimha G R
      2. YARN-2934.v1.002.patch
        21 kB
        Naganarasimha G R
      3. YARN-2934.v1.003.patch
        18 kB
        Naganarasimha G R
      4. YARN-2934.v1.004.patch
        19 kB
        Naganarasimha G R
      5. YARN-2934.v1.005.patch
        22 kB
        Naganarasimha G R
      6. YARN-2934.v1.006.patch
        23 kB
        Naganarasimha G R
      7. YARN-2934.v1.007.patch
        23 kB
        Naganarasimha G R
      8. YARN-2934.v1.008.patch
        23 kB
        Naganarasimha G R
      9. YARN-2934.v2.001.patch
        20 kB
        Naganarasimha G R
      10. YARN-2934.v2.002.patch
        19 kB
        Naganarasimha G R
      11. YARN-2934.v2.003.patch
        21 kB
        Naganarasimha G R
      12. YARN-2934.v2.004.patch
        21 kB
        Naganarasimha G R

        Issue Links

          Activity

          Hide
          jira.shegalov Gera Shegalov added a comment -

          We need to make sure that stderr location is made known in the container launch context such that the wrapper script can cat it to it's stderr and it can be consumed by Shell

          Show
          jira.shegalov Gera Shegalov added a comment - We need to make sure that stderr location is made known in the container launch context such that the wrapper script can cat it to it's stderr and it can be consumed by Shell
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Gera Shegalov,
          AFAIK some time launching container fails and the exit information is present in the NM logs

          2014-12-06 14:43:42,583 | WARN | ContainersLauncher #32 | Exception from container-launch with container ID: container_1417695705818_0094_01_33554435 and exit code: 134 | org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:310)
          ExitCodeException exitCode=134: /bin/bash: line 1: 77143 Aborted /opt/huawei/Bigdata/jdk1.7.0_60//bin/java -Xms2048m -Xmx2048m -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128M -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=20 -XX:SoftRefLRUPolicyMSPerMB=0 -agentlib:jdwp=transport=dt_socket,address=33233,server=y,suspend=n com.huawei.data.integration.dataflow.runtime.yarn.app.StreamNodeContainer -cc-host linux81 -cc-port 32335 -cluster-net-ip-address linux81 -data-ip-address linux81 -node-id linux81 -result-ip-address linux81 > /srv/BigData/yarn/nm/containerlogs/application_1417695705818_0094/container_1417695705818_0094_01_33554435/stdout 2> /srv/BigData/yarn/nm/containerlogs/application_1417695705818_0094/container_1417695705818_0094_01_33554435/stderr

          at org.apache.hadoop.util.Shell.runCommand(Shell.java:537)
          at org.apache.hadoop.util.Shell.run(Shell.java:454)

          And these kind of errors/exceptions needs to be present in container's stderr also . Is this the goal of this jira ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Gera Shegalov , AFAIK some time launching container fails and the exit information is present in the NM logs 2014-12-06 14:43:42,583 | WARN | ContainersLauncher #32 | Exception from container-launch with container ID: container_1417695705818_0094_01_33554435 and exit code: 134 | org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:310) ExitCodeException exitCode=134: /bin/bash: line 1: 77143 Aborted /opt/huawei/Bigdata/jdk1.7.0_60//bin/java -Xms2048m -Xmx2048m -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128M -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=20 -XX:SoftRefLRUPolicyMSPerMB=0 -agentlib:jdwp=transport=dt_socket,address=33233,server=y,suspend=n com.huawei.data.integration.dataflow.runtime.yarn.app.StreamNodeContainer -cc-host linux81 -cc-port 32335 -cluster-net-ip-address linux81 -data-ip-address linux81 -node-id linux81 -result-ip-address linux81 > /srv/BigData/yarn/nm/containerlogs/application_1417695705818_0094/container_1417695705818_0094_01_33554435/stdout 2> /srv/BigData/yarn/nm/containerlogs/application_1417695705818_0094/container_1417695705818_0094_01_33554435/stderr at org.apache.hadoop.util.Shell.runCommand(Shell.java:537) at org.apache.hadoop.util.Shell.run(Shell.java:454) And these kind of errors/exceptions needs to be present in container's stderr also . Is this the goal of this jira ?
          Hide
          xgong Xuan Gong added a comment -

          Hey, Gera Shegalov
          we already have a Environment.LOG_DIRS env which returns a comma separated list of log-dirs.

          Show
          xgong Xuan Gong added a comment - Hey, Gera Shegalov we already have a Environment.LOG_DIRS env which returns a comma separated list of log-dirs.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Naganarasimha G R, yes that's what I meant. Maybe this is specific to the DefaultContainerExecutor.
          When testing on my macbook:

          $ hadoop org.apache.hadoop.mapreduce.SleepJob -Dyarn.app.mapreduce.am.env=JAVA_HOME=/no/jvm/here -m 1
          

          All you get:

          2014-12-09 09:15:00,252 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1418144997824_0001_01_000001 and exit code: 127
          ExitCodeException exitCode=127:
                  at org.apache.hadoop.util.Shell.runCommand(Shell.java:544)
                  at org.apache.hadoop.util.Shell.run(Shell.java:455)
                  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:721)
                  at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
                  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
                  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                  at java.lang.Thread.run(Thread.java:745)
          

          In the stderr log of the container, you can see the real deal:

          Log Type: stderr
          Log Upload Time: Tue Dec 09 09:15:05 -0800 2014
          Log Length: 60
          /bin/bash: /no/jvm/here/bin/java: No such file or directory
          
          Show
          jira.shegalov Gera Shegalov added a comment - Hi Naganarasimha G R , yes that's what I meant. Maybe this is specific to the DefaultContainerExecutor . When testing on my macbook: $ hadoop org.apache.hadoop.mapreduce.SleepJob -Dyarn.app.mapreduce.am.env=JAVA_HOME=/no/jvm/here -m 1 All you get: 2014-12-09 09:15:00,252 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1418144997824_0001_01_000001 and exit code: 127 ExitCodeException exitCode=127: at org.apache.hadoop.util.Shell.runCommand(Shell.java:544) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:721) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) In the stderr log of the container, you can see the real deal: Log Type: stderr Log Upload Time: Tue Dec 09 09:15:05 -0800 2014 Log Length: 60 /bin/bash: /no/jvm/here/bin/java: No such file or directory
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Xuan Gong,

          we already have a Environment.LOG_DIRS env which returns a comma separated list of log-dirs.

          I am aware of this. However, each app is free to pump their standard error "2>" in any file under this dir.

          Show
          jira.shegalov Gera Shegalov added a comment - Hi Xuan Gong , we already have a Environment.LOG_DIRS env which returns a comma separated list of log-dirs. I am aware of this. However, each app is free to pump their standard error "2>" in any file under this dir.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Gera Shegalov,
          Had a look at the code, As you mentioned each app is free to pump error stream to any file and each app sends the command (as part of ContainerLaunchContext) and sets the out and err file in it like "<LOG_DIR>"/stdout and "<LOG_DIR>"/syserr. Later ContainerLaunch will take care of expanding the env variables before writing it into a script file which is further read/utilized by the container executors.
          AFAIU, in the approach which you have suggested you are expecting apps to set the stderr file in ContainerLaunchContext so that if there is any error then the wrapper script (i.e. ContainerLaunchContext) of DefaultContainerExecutor can cat it and error is captured by ShellExecutor. But this solves only for DefaultContainerExecutor any thoughts for handling in the case of Linux containerExecutor ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Gera Shegalov , Had a look at the code, As you mentioned each app is free to pump error stream to any file and each app sends the command (as part of ContainerLaunchContext) and sets the out and err file in it like "<LOG_DIR>"/stdout and "<LOG_DIR>"/syserr. Later ContainerLaunch will take care of expanding the env variables before writing it into a script file which is further read/utilized by the container executors. AFAIU, in the approach which you have suggested you are expecting apps to set the stderr file in ContainerLaunchContext so that if there is any error then the wrapper script (i.e. ContainerLaunchContext) of DefaultContainerExecutor can cat it and error is captured by ShellExecutor. But this solves only for DefaultContainerExecutor any thoughts for handling in the case of Linux containerExecutor ?
          Hide
          jira.shegalov Gera Shegalov added a comment -

          ContainerLaunchContext is meant for a ContainerExecutor in general. In the LCE case the logs may only be readable by the app user. To make it robust we can simply append the catting to the supplied command that runs in the executor. This is the hacky version for it disregarding OS diversity. It presumes that stderr log has "strderr" in the file name.

          $ git diff
          diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          index a87238d..8ea2560 100644
          --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          @@ -190,6 +190,11 @@ public Integer call() {
                   // TODO: Should we instead work via symlinks without this grammar?
                   newCmds.add(expandEnvironment(str, containerLogDir));
                 }
          +      newCmds.add("||");
          +      newCmds.add("(cat " + containerLogDir + "/*stderr* 1>&2");
          +      newCmds.add(";");
          +      newCmds.add("exit -1)");
          +
                 launchContext.setCommands(newCmds);
           
                 Map<String, String> environment = launchContext.getEnvironment();
          

          Then we get the desired effect:

          ]$ hadoop org.apache.hadoop.mapreduce.SleepJob -Dyarn.app.mapreduce.am.env=JAVA_HOME=/no/jvm/here -m 10 
          15/01/06 23:36:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
          15/01/06 23:36:14 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
          15/01/06 23:36:15 INFO mapreduce.JobSubmitter: number of splits:10
          15/01/06 23:36:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1420533216615_0013
          15/01/06 23:36:16 INFO impl.YarnClientImpl: Submitted application application_1420533216615_0013
          15/01/06 23:36:16 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1420533216615_0013/
          15/01/06 23:36:16 INFO mapreduce.Job: Running job: job_1420533216615_0013
          15/01/06 23:36:21 INFO mapreduce.Job: Job job_1420533216615_0013 running in uber mode : false
          15/01/06 23:36:21 INFO mapreduce.Job:  map 0% reduce 0%
          15/01/06 23:36:21 INFO mapreduce.Job: Job job_1420533216615_0013 failed with state FAILED due to: Application application_1420533216615_0013 failed 2 times due to AM Container for appattempt_1420533216615_0013_000002 exited with  exitCode: 255
          For more detailed output, check application tracking page:http://localhost:8088/proxy/application_1420533216615_0013/Then, click on links to logs of each attempt.
          Diagnostics: Exception from container-launch.
          Container id: container_1420533216615_0013_02_000001
          Exit code: 255
          Exception message: /bin/bash: /no/jvm/here/bin/java: No such file or directory
          
          Stack trace: ExitCodeException exitCode=255: /bin/bash: /no/jvm/here/bin/java: No such file or directory
          
          	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
          	at org.apache.hadoop.util.Shell.run(Shell.java:456)
          	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
          	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
          	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:307)
          	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          
          Show
          jira.shegalov Gera Shegalov added a comment - ContainerLaunchContext is meant for a ContainerExecutor in general. In the LCE case the logs may only be readable by the app user. To make it robust we can simply append the catting to the supplied command that runs in the executor. This is the hacky version for it disregarding OS diversity. It presumes that stderr log has "strderr" in the file name. $ git diff diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java index a87238d..8ea2560 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java @@ -190,6 +190,11 @@ public Integer call() { // TODO: Should we instead work via symlinks without this grammar? newCmds.add(expandEnvironment(str, containerLogDir)); } + newCmds.add( "||" ); + newCmds.add( "(cat " + containerLogDir + "/*stderr* 1>&2" ); + newCmds.add( ";" ); + newCmds.add( "exit -1)" ); + launchContext.setCommands(newCmds); Map< String , String > environment = launchContext.getEnvironment(); Then we get the desired effect: ]$ hadoop org.apache.hadoop.mapreduce.SleepJob -Dyarn.app.mapreduce.am.env=JAVA_HOME=/no/jvm/here -m 10 15/01/06 23:36:13 WARN util.NativeCodeLoader: Unable to load native -hadoop library for your platform... using builtin-java classes where applicable 15/01/06 23:36:14 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032 15/01/06 23:36:15 INFO mapreduce.JobSubmitter: number of splits:10 15/01/06 23:36:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1420533216615_0013 15/01/06 23:36:16 INFO impl.YarnClientImpl: Submitted application application_1420533216615_0013 15/01/06 23:36:16 INFO mapreduce.Job: The url to track the job: http: //localhost:8088/proxy/application_1420533216615_0013/ 15/01/06 23:36:16 INFO mapreduce.Job: Running job: job_1420533216615_0013 15/01/06 23:36:21 INFO mapreduce.Job: Job job_1420533216615_0013 running in uber mode : false 15/01/06 23:36:21 INFO mapreduce.Job: map 0% reduce 0% 15/01/06 23:36:21 INFO mapreduce.Job: Job job_1420533216615_0013 failed with state FAILED due to: Application application_1420533216615_0013 failed 2 times due to AM Container for appattempt_1420533216615_0013_000002 exited with exitCode: 255 For more detailed output, check application tracking page:http: //localhost:8088/proxy/application_1420533216615_0013/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1420533216615_0013_02_000001 Exit code: 255 Exception message: /bin/bash: /no/jvm/here/bin/java: No such file or directory Stack trace: ExitCodeException exitCode=255: /bin/bash: /no/jvm/here/bin/java: No such file or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:307) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745)
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          This seems like it is same as YARN-2013/YARN-2242.

          For the AM, we ended up simply putting a log-message "AM Container for \$ApplicationAttemptId exited with exitCode: \$ExitStatus.\nFor more detailed output, check application tracking page: \$TrackingUrl, Then, click on links to logs of each attempt.\n"

          You really don't want to "cat" stderr from containers. Containers may run for a very long time, spewing a lot of errors in stderr before finally failing. NM unconditionally reading logs in such cases will blow up NM heap. We either do a cross-platform way of tailing the last N bytes (not terribly useful if we cut lines mid way through) or better simply print a link to take them to the right set of logs.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - This seems like it is same as YARN-2013 / YARN-2242 . For the AM, we ended up simply putting a log-message "AM Container for \$ApplicationAttemptId exited with exitCode: \$ExitStatus.\nFor more detailed output, check application tracking page: \$TrackingUrl, Then, click on links to logs of each attempt.\n" You really don't want to "cat" stderr from containers. Containers may run for a very long time, spewing a lot of errors in stderr before finally failing. NM unconditionally reading logs in such cases will blow up NM heap. We either do a cross-platform way of tailing the last N bytes (not terribly useful if we cut lines mid way through) or better simply print a link to take them to the right set of logs.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Yes it's related, but not exclusive to AM (try -Dmapreduce.map.env=JAVA_HOME=/no/jvm/here). It's just more severe with AM. "cat" is not the point. Getting the real diagnostics with something is, +1 for using tail. The pointer to the tracking page can be of little value for a busy cluster. The RMApp is likely to age out by the time the user gets to look at it, and there is no JHS entry because the AM crashed. It would be better to mention the nodeAddress as well, in addition to containerId to be used with 'yarn logs'

          Show
          jira.shegalov Gera Shegalov added a comment - Yes it's related, but not exclusive to AM (try -Dmapreduce.map.env=JAVA_HOME=/no/jvm/here). It's just more severe with AM. "cat" is not the point. Getting the real diagnostics with something is, +1 for using tail. The pointer to the tracking page can be of little value for a busy cluster. The RMApp is likely to age out by the time the user gets to look at it, and there is no JHS entry because the AM crashed. It would be better to mention the nodeAddress as well, in addition to containerId to be used with 'yarn logs'
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Yes it's related, but not exclusive to AM (try -Dmapreduce.map.env=JAVA_HOME=/no/jvm/here). It's just more severe with AM.

          Agreed. I was just saying that we can do what we did for AMs.

          The pointer to the tracking page can be of little value for a busy cluster. The RMApp is likely to age out by the time the user gets to look at it, and there is no JHS entry because the AM crashed.

          Good point, I missed this one. Given this, even the tailed stderr is not useful in such a situation. If the app-page ages out, where will the user see this additional diagnostic message that we tail out of logs?

          It would be better to mention the nodeAddress as well, in addition to containerId to be used with 'yarn logs'

          This can be done in the additional message (like for AM) instead of cat/tail of logs.

          I guess the options are (1) Diagnostic message with links and reference to the right logs saying something happened or (2) Diagnostic message itself containing the tail of the log (which may or may not really determine the error message). I think (1) is a must, (2) is a good to have.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Yes it's related, but not exclusive to AM (try -Dmapreduce.map.env=JAVA_HOME=/no/jvm/here). It's just more severe with AM. Agreed. I was just saying that we can do what we did for AMs. The pointer to the tracking page can be of little value for a busy cluster. The RMApp is likely to age out by the time the user gets to look at it, and there is no JHS entry because the AM crashed. Good point, I missed this one. Given this, even the tailed stderr is not useful in such a situation. If the app-page ages out, where will the user see this additional diagnostic message that we tail out of logs? It would be better to mention the nodeAddress as well, in addition to containerId to be used with 'yarn logs' This can be done in the additional message (like for AM) instead of cat/tail of logs. I guess the options are (1) Diagnostic message with links and reference to the right logs saying something happened or (2) Diagnostic message itself containing the tail of the log (which may or may not really determine the error message). I think (1) is a must, (2) is a good to have.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Given this, even the tailed stderr is not useful in such a situation. If the app-page ages out, where will the user see this additional diagnostic message that we tail out of logs?

          It will be in the client output that I showed in the above comments. In our infrastructure, a failed job will generate an alert email containing the client log (or link to it).

          Show
          jira.shegalov Gera Shegalov added a comment - Given this, even the tailed stderr is not useful in such a situation. If the app-page ages out, where will the user see this additional diagnostic message that we tail out of logs? It will be in the client output that I showed in the above comments. In our infrastructure, a failed job will generate an alert email containing the client log (or link to it).
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Gera Shegalov,
          Sorry for the long gap in handling this, but had a query related to this, tail can only work in case of linux systems so was wondering how to keep it as a neutral implementation ? May be RandomAccessFile, thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Gera Shegalov , Sorry for the long gap in handling this, but had a query related to this, tail can only work in case of linux systems so was wondering how to keep it as a neutral implementation ? May be RandomAccessFile, thoughts ?
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Naganarasimha G R, yes I was thinking the same, we should try to do it in the java land. I'd prefer using RawLocalFileSytem#read(buf, off, len) in order not to mix in java.io API. Since the NM webUI can read logs, we should have no problems accessing them from the NM JVM.

          Show
          jira.shegalov Gera Shegalov added a comment - Hi Naganarasimha G R , yes I was thinking the same, we should try to do it in the java land. I'd prefer using RawLocalFileSytem#read(buf, off, len) in order not to mix in java.io API. Since the NM webUI can read logs, we should have no problems accessing them from the NM JVM.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Gera Shegalov,
          Apologies from my side, missed to update on this jira, thanks nijel for pushing me on this !
          I have attached an initial version of the patch which suffices but following points are open for discussion

          1. Errorfilename is sent from the application/client hence name can be any thing. Currently have used WildCard Matcher on fileName and the pattern used is "stderr" and also made it configurable. If the admin wants to attach multiple types then Regex matching would be required, but regex pattern is not similar to unix pattern hence for the sake of simplicity have kept Wildcard. Thoughts?
          2. Or other way round may be i can support as you were mentioning like, support ApplicationSubmissionContext to expose interface for clients to inform the error filename and on error use this name directly if not present then use the pattern matching. But my concern is what if the error file name mentioned as part of interface is not matching the execution command sent !.
          3. Also Bikas Saha mentioned in his comment, do i need to additionally check for syslog also? if so only if the syserr doesnt exist ? and similarly what approaches would be ideal for identifying the name of the log4j file?
          4. have hard coded tail size to be 4k, is it required to be configurable ?
          Show
          Naganarasimha Naganarasimha G R added a comment - Gera Shegalov , Apologies from my side, missed to update on this jira, thanks nijel for pushing me on this ! I have attached an initial version of the patch which suffices but following points are open for discussion Errorfilename is sent from the application/client hence name can be any thing. Currently have used WildCard Matcher on fileName and the pattern used is " stderr " and also made it configurable. If the admin wants to attach multiple types then Regex matching would be required, but regex pattern is not similar to unix pattern hence for the sake of simplicity have kept Wildcard. Thoughts? Or other way round may be i can support as you were mentioning like, support ApplicationSubmissionContext to expose interface for clients to inform the error filename and on error use this name directly if not present then use the pattern matching. But my concern is what if the error file name mentioned as part of interface is not matching the execution command sent !. Also Bikas Saha mentioned in his comment , do i need to additionally check for syslog also? if so only if the syserr doesnt exist ? and similarly what approaches would be ideal for identifying the name of the log4j file? have hard coded tail size to be 4k, is it required to be configurable ?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 3m 13s trunk passed
          +1 compile 0m 57s trunk passed with JDK v1.8.0_66
          +1 compile 0m 51s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 28s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 2m 19s trunk passed
          +1 javadoc 0m 59s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 29s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 56s the patch passed with JDK v1.8.0_66
          +1 javac 0m 56s the patch passed
          +1 compile 0m 52s the patch passed with JDK v1.7.0_79
          +1 javac 0m 52s the patch passed
          -1 checkstyle 0m 29s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 277, now 277).
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager introduced 1 new FindBugs issues.
          +1 javadoc 1m 0s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 19s the patch passed with JDK v1.7.0_79
          -1 unit 0m 23s hadoop-yarn-api in the patch failed with JDK v1.8.0_66.
          +1 unit 8m 52s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          -1 unit 0m 23s hadoop-yarn-api in the patch failed with JDK v1.7.0_79.
          +1 unit 9m 15s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          44m 5s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
            Found reliance on default encoding in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorLogTail(Path):in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorLogTail(Path): new String(byte[]) At ContainerLaunch.java:[line 426]
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
          JDK v1.7.0_79 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-10-31
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769927/YARN-2934.v1.001.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 456e709b8107 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / a4a6b5b
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9609/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9609/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 3m 13s trunk passed +1 compile 0m 57s trunk passed with JDK v1.8.0_66 +1 compile 0m 51s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 28s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 2m 19s trunk passed +1 javadoc 0m 59s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 29s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 50s the patch passed +1 compile 0m 56s the patch passed with JDK v1.8.0_66 +1 javac 0m 56s the patch passed +1 compile 0m 52s the patch passed with JDK v1.7.0_79 +1 javac 0m 52s the patch passed -1 checkstyle 0m 29s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 277, now 277). +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager introduced 1 new FindBugs issues. +1 javadoc 1m 0s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 19s the patch passed with JDK v1.7.0_79 -1 unit 0m 23s hadoop-yarn-api in the patch failed with JDK v1.8.0_66. +1 unit 8m 52s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. -1 unit 0m 23s hadoop-yarn-api in the patch failed with JDK v1.7.0_79. +1 unit 9m 15s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 44m 5s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   Found reliance on default encoding in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorLogTail(Path):in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorLogTail(Path): new String(byte[]) At ContainerLaunch.java: [line 426] JDK v1.8.0_66 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields JDK v1.7.0_79 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-10-31 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769927/YARN-2934.v1.001.patch JIRA Issue YARN-2934 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 456e709b8107 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / a4a6b5b Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9609/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9609/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9609/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          good first pass.

          I wouldn't worry about a configurable size for now (who gets to configure it?), but I would derive the text "4k bytes" from the string constant, instead of duplicate

          needs a test; some minicluster one will suffice

          Show
          stevel@apache.org Steve Loughran added a comment - good first pass. I wouldn't worry about a configurable size for now (who gets to configure it?), but I would derive the text "4k bytes" from the string constant, instead of duplicate needs a test; some minicluster one will suffice
          Hide
          varun_saxena Varun Saxena added a comment -

          Thanks Naganarasimha G R for updating the patch. In general , the approach looks fine. Few comments :

          1. FSDataInputStream in ContainerLaunch#getErrorLogTail is not closed. You can use try with resources construct or try-finally construct to close it. Latter if this has to go in branch-2.6 as well
          2. The newly added configuration should be added in yarn-default.xml as well.
          Show
          varun_saxena Varun Saxena added a comment - Thanks Naganarasimha G R for updating the patch. In general , the approach looks fine. Few comments : FSDataInputStream in ContainerLaunch#getErrorLogTail is not closed. You can use try with resources construct or try-finally construct to close it. Latter if this has to go in branch-2.6 as well The newly added configuration should be added in yarn-default.xml as well.
          Hide
          varun_saxena Varun Saxena added a comment -

          One more comment :
          FSDataInputStream#read is not guaranteed to read all the bytes which you have passed. You have to check number of bytes read and loop until read returns -1.

          Show
          varun_saxena Varun Saxena added a comment - One more comment : FSDataInputStream#read is not guaranteed to read all the bytes which you have passed. You have to check number of bytes read and loop until read returns -1.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the review comments Steve Loughran & Varun Saxena,
          Had kept test and the xml update to be in later patches as i thought of getting the feedback first, As i have received some comments and unit tests failing, i have updated them in the patch and with fixes for other review comments

          some minicluster one will suffice

          Had anyway started of with mocking required in ContainerExecutor, hence went ahead with it. Hope it would not be an issue, please inform if otherwise

          try-finally construct to close it. Latter if this has to go in branch-2.6 as well

          As i was not sure whether this jira will be used in 2.6, hence used normal try-finally construct to close it.

          FSDataInputStream#read is not guaranteed to read all the bytes which you have passed. You have to check number of bytes read and loop until read returns -1.

          There was no explicit mention that if the length of the buffer is sufficiently given then also it might not read all the contents of the file in to the buffer in the read call, but checked with Vinayakumar B and he confirmed the above is possible and hence used FSDataInputStream.readFully()

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review comments Steve Loughran & Varun Saxena , Had kept test and the xml update to be in later patches as i thought of getting the feedback first, As i have received some comments and unit tests failing, i have updated them in the patch and with fixes for other review comments some minicluster one will suffice Had anyway started of with mocking required in ContainerExecutor, hence went ahead with it. Hope it would not be an issue, please inform if otherwise try-finally construct to close it. Latter if this has to go in branch-2.6 as well As i was not sure whether this jira will be used in 2.6, hence used normal try-finally construct to close it. FSDataInputStream#read is not guaranteed to read all the bytes which you have passed. You have to check number of bytes read and loop until read returns -1. There was no explicit mention that if the length of the buffer is sufficiently given then also it might not read all the contents of the file in to the buffer in the read call, but checked with Vinayakumar B and he confirmed the above is possible and hence used FSDataInputStream.readFully()
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 3m 49s trunk passed
          +1 compile 0m 58s trunk passed with JDK v1.8.0_60
          +1 compile 0m 52s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 29s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          -1 findbugs 1m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 21s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 51s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 15s the patch passed
          +1 compile 0m 49s the patch passed with JDK v1.8.0_60
          +1 javac 0m 49s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.7.0_79
          +1 javac 0m 47s the patch passed
          -1 checkstyle 0m 23s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 277, now 278).
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 3m 56s the patch passed
          +1 javadoc 1m 21s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 49s the patch passed with JDK v1.7.0_79
          +1 unit 0m 21s hadoop-yarn-api in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 50s hadoop-yarn-common in the patch passed with JDK v1.8.0_60.
          -1 unit 7m 29s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_60.
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_79.
          +1 unit 2m 3s hadoop-yarn-common in the patch passed with JDK v1.7.0_79.
          -1 unit 7m 56s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_79.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          50m 36s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
            hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
            hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
            hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
            hadoop.yarn.server.nodemanager.containermanager.TestNMProxy
          JDK v1.7.0_79 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
            hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
            hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
            hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
            hadoop.yarn.server.nodemanager.containermanager.TestNMProxy



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-02
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770061/YARN-2934.v1.002.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml
          uname Linux 5916f515832b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / 6e4f8a4
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9611/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9611/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 49s trunk passed +1 compile 0m 58s trunk passed with JDK v1.8.0_60 +1 compile 0m 52s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 29s trunk passed +1 mvneclipse 0m 44s trunk passed -1 findbugs 1m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 21s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 51s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 15s the patch passed +1 compile 0m 49s the patch passed with JDK v1.8.0_60 +1 javac 0m 49s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_79 +1 javac 0m 47s the patch passed -1 checkstyle 0m 23s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 277, now 278). +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 56s the patch passed +1 javadoc 1m 21s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 49s the patch passed with JDK v1.7.0_79 +1 unit 0m 21s hadoop-yarn-api in the patch passed with JDK v1.8.0_60. +1 unit 1m 50s hadoop-yarn-common in the patch passed with JDK v1.8.0_60. -1 unit 7m 29s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_60. +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_79. +1 unit 2m 3s hadoop-yarn-common in the patch passed with JDK v1.7.0_79. -1 unit 7m 56s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_79. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 50m 36s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager   hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor   hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch   hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService   hadoop.yarn.server.nodemanager.containermanager.TestNMProxy JDK v1.7.0_79 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager   hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor   hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch   hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService   hadoop.yarn.server.nodemanager.containermanager.TestNMProxy Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-02 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770061/YARN-2934.v1.002.patch JIRA Issue YARN-2934 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml uname Linux 5916f515832b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / 6e4f8a4 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9611/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9611/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          fixing checkstyle and test cases !

          Show
          Naganarasimha Naganarasimha G R added a comment - fixing checkstyle and test cases !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 14s trunk passed
          +1 compile 0m 47s trunk passed with JDK v1.8.0_60
          +1 compile 0m 46s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 24s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 21s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 46s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 15s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.8.0_60
          +1 javac 0m 47s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.7.0_79
          +1 javac 0m 47s the patch passed
          -1 checkstyle 0m 24s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 277, now 277).
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 53s the patch passed
          +1 javadoc 1m 24s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 56s the patch passed with JDK v1.7.0_79
          +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 53s hadoop-yarn-common in the patch passed with JDK v1.8.0_60.
          -1 unit 9m 3s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_60.
          +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.7.0_79.
          +1 unit 2m 3s hadoop-yarn-common in the patch passed with JDK v1.7.0_79.
          +1 unit 9m 6s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          52m 12s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-02
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770076/YARN-2934.v1.003.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml
          uname Linux 70a5f2469f5d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / 6e4f8a4
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9612/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9612/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9612/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9612/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9612/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9612/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 14s trunk passed +1 compile 0m 47s trunk passed with JDK v1.8.0_60 +1 compile 0m 46s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 24s trunk passed +1 mvneclipse 0m 39s trunk passed -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 21s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 46s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 15s the patch passed +1 compile 0m 47s the patch passed with JDK v1.8.0_60 +1 javac 0m 47s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_79 +1 javac 0m 47s the patch passed -1 checkstyle 0m 24s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 277, now 277). +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 53s the patch passed +1 javadoc 1m 24s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 56s the patch passed with JDK v1.7.0_79 +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.8.0_60. +1 unit 1m 53s hadoop-yarn-common in the patch passed with JDK v1.8.0_60. -1 unit 9m 3s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_60. +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.7.0_79. +1 unit 2m 3s hadoop-yarn-common in the patch passed with JDK v1.7.0_79. +1 unit 9m 6s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 52m 12s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-02 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770076/YARN-2934.v1.003.patch JIRA Issue YARN-2934 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml uname Linux 70a5f2469f5d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / 6e4f8a4 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9612/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9612/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9612/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9612/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9612/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9612/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Gera Shegalov, Steve Loughran & Varun Saxena,
          Manually ran for the findbugs report on NodeManager project and seems to be fine. And checkstyle issue is insignificant.
          Hope to get some review on this !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Gera Shegalov , Steve Loughran & Varun Saxena , Manually ran for the findbugs report on NodeManager project and seems to be fine. And checkstyle issue is insignificant. Hope to get some review on this !
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Can one of the watchers, please take a look at the patch ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Can one of the watchers, please take a look at the patch ?
          Hide
          nijel nijel added a comment -

          thanks Naganarasimha G R for the patch

          Few minor comments/doubts

          1.

          FileStatus[] listStatus =
                 fileSystem.listStatus(containerLogDir, new PathFilter() {
                   @Override
                   public boolean accept(Path path) {
                     return FilenameUtils.wildcardMatch(path.getName(),
                         errorFileNamePattern, IOCase.INSENSITIVE);
                   }
                 });
          

          What if this give multiple error files ?

          2.

           } catch (IOException e) {
                  LOG.warn("Failed while trying to read container's error log", e);
                }
          

          Can this be error log ? I think there should not be any exception in reading the file. If there is an error, better to log error log

          Show
          nijel nijel added a comment - thanks Naganarasimha G R for the patch Few minor comments/doubts 1. FileStatus[] listStatus = fileSystem.listStatus(containerLogDir, new PathFilter() { @Override public boolean accept(Path path) { return FilenameUtils.wildcardMatch(path.getName(), errorFileNamePattern, IOCase.INSENSITIVE); } }); What if this give multiple error files ? 2. } catch (IOException e) { LOG.warn( "Failed while trying to read container's error log" , e); } Can this be error log ? I think there should not be any exception in reading the file. If there is an error, better to log error log
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the comments nijel,

          What if this give multiple error files ?

          Currenly i am going under the assumption that the logs needs to be only fetched from stderr and hence there is less possibility of multiple files, but if app is submitted wrongly configured file names then the first file matching the pattern is picked.

          Other comment for error message is corrected, also discussed with Rohith Sharma K S, we felt that Regex is better as we can specify multiple patterns in a single pattern which can help in future multiple apps scenario.
          Uploading a patch With Regex Pattern approach

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the comments nijel , What if this give multiple error files ? Currenly i am going under the assumption that the logs needs to be only fetched from stderr and hence there is less possibility of multiple files, but if app is submitted wrongly configured file names then the first file matching the pattern is picked. Other comment for error message is corrected, also discussed with Rohith Sharma K S , we felt that Regex is better as we can specify multiple patterns in a single pattern which can help in future multiple apps scenario. Uploading a patch With Regex Pattern approach
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 2s trunk passed
          +1 compile 0m 50s trunk passed with JDK v1.8.0_60
          +1 compile 0m 46s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 27s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          -1 findbugs 1m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 23s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 50s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 17s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.8.0_60
          +1 javac 0m 47s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.7.0_79
          +1 javac 0m 47s the patch passed
          -1 checkstyle 0m 26s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 277, now 278).
          +1 mvneclipse 0m 37s the patch passed
          -1 whitespace 0m 0s The patch has 9 line(s) with tabs.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 55s the patch passed
          +1 javadoc 1m 25s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 51s the patch passed with JDK v1.7.0_79
          +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 52s hadoop-yarn-common in the patch passed with JDK v1.8.0_60.
          -1 unit 8m 30s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_60.
          +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_79.
          +1 unit 2m 3s hadoop-yarn-common in the patch passed with JDK v1.7.0_79.
          +1 unit 9m 5s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          51m 39s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-09
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771303/YARN-2934.v1.004.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml
          uname Linux 179283f3ad1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / 668e897
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9642/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9642/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 2s trunk passed +1 compile 0m 50s trunk passed with JDK v1.8.0_60 +1 compile 0m 46s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 27s trunk passed +1 mvneclipse 0m 38s trunk passed -1 findbugs 1m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 23s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 50s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 17s the patch passed +1 compile 0m 47s the patch passed with JDK v1.8.0_60 +1 javac 0m 47s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_79 +1 javac 0m 47s the patch passed -1 checkstyle 0m 26s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 277, now 278). +1 mvneclipse 0m 37s the patch passed -1 whitespace 0m 0s The patch has 9 line(s) with tabs. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 55s the patch passed +1 javadoc 1m 25s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 51s the patch passed with JDK v1.7.0_79 +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_60. +1 unit 1m 52s hadoop-yarn-common in the patch passed with JDK v1.8.0_60. -1 unit 8m 30s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_60. +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_79. +1 unit 2m 3s hadoop-yarn-common in the patch passed with JDK v1.7.0_79. +1 unit 9m 5s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 51m 39s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-09 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771303/YARN-2934.v1.004.patch JIRA Issue YARN-2934 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml uname Linux 179283f3ad1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / 668e897 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_60.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9642/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9642/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Findbugs is not related to this jira and check style & white space issue can be corrected as part of the next patch, Waiting for review comments ! cc/ Gera Shegalov & Bikas Saha.

          Show
          Naganarasimha Naganarasimha G R added a comment - Findbugs is not related to this jira and check style & white space issue can be corrected as part of the next patch, Waiting for review comments ! cc/ Gera Shegalov & Bikas Saha .
          Hide
          varun_saxena Varun Saxena added a comment -

          Thanks Naganarasimha G R for uploading the patch. Sorry could not do a thorough review earlier.
          Had a cursory glance at the latest patch. A few quick comments.

          • In the code below, instead of compiling the pattern again and again, we can compile it once and store it in a static variable(because its taken from config and hence wont change).
            Pattern#compile incurs a performance overhead if called again and again.
                String errorFileNameRegexPattern =
                   conf.get(YarnConfiguration.NM_CONTAINER_ERROR_FILE_NAME_PATTERN,
            	     YarnConfiguration.DEFAULT_NM_CONTAINER_ERROR_FILE_NAME_PATTERN);
                Pattern pattern = null;
                try {
                  pattern =
                      Pattern.compile(errorFileNameRegexPattern, Pattern.CASE_INSENSITIVE);
                } catch (PatternSyntaxException e) {
                  pattern = Pattern.compile(
                      YarnConfiguration.DEFAULT_NM_CONTAINER_ERROR_FILE_NAME_PATTERN,
                      Pattern.CASE_INSENSITIVE);
               }
            
          • Also IMO, atleast a warning log should be printed if configured pattern cannot compile. This can alert the user about wrong configuration.
            Should we consider not starting up NM in this case(if config is wrong) ? Maybe its not that important a config to not start NM. An alert message should be enough.
          • Moreover, you can also consider using Configuration#getPattern, but take care of using it only once.
          Show
          varun_saxena Varun Saxena added a comment - Thanks Naganarasimha G R for uploading the patch. Sorry could not do a thorough review earlier. Had a cursory glance at the latest patch. A few quick comments. In the code below, instead of compiling the pattern again and again , we can compile it once and store it in a static variable(because its taken from config and hence wont change). Pattern#compile incurs a performance overhead if called again and again. String errorFileNameRegexPattern = conf.get(YarnConfiguration.NM_CONTAINER_ERROR_FILE_NAME_PATTERN, YarnConfiguration.DEFAULT_NM_CONTAINER_ERROR_FILE_NAME_PATTERN); Pattern pattern = null ; try { pattern = Pattern.compile(errorFileNameRegexPattern, Pattern.CASE_INSENSITIVE); } catch (PatternSyntaxException e) { pattern = Pattern.compile( YarnConfiguration.DEFAULT_NM_CONTAINER_ERROR_FILE_NAME_PATTERN, Pattern.CASE_INSENSITIVE); } Also IMO, atleast a warning log should be printed if configured pattern cannot compile. This can alert the user about wrong configuration. Should we consider not starting up NM in this case(if config is wrong) ? Maybe its not that important a config to not start NM. An alert message should be enough. Moreover, you can also consider using Configuration#getPattern, but take care of using it only once.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the comments Varun Saxena,

          Pattern#compile incurs a performance overhead if called again and again.

          thought of handling this based on further comments and also it was not in critical/repititive code flow but any way worth optimizing hence have done in this patch.

          Should we consider not starting up NM in this case(if config is wrong) ? Maybe its not that important a config to not start NM. An alert message should be enough.

          As discussed alert is enough as its not critical.

          Moreover, you can also consider using Configuration#getPattern, but take care of using it only once.

          Yep this would be usefull, and also takes care of your 2nd comment, hence using this. but adding one more additional method there to ignore the case.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the comments Varun Saxena , Pattern#compile incurs a performance overhead if called again and again. thought of handling this based on further comments and also it was not in critical/repititive code flow but any way worth optimizing hence have done in this patch. Should we consider not starting up NM in this case(if config is wrong) ? Maybe its not that important a config to not start NM. An alert message should be enough. As discussed alert is enough as its not critical. Moreover, you can also consider using Configuration#getPattern, but take care of using it only once. Yep this would be usefull, and also takes care of your 2nd comment, hence using this. but adding one more additional method there to ignore the case.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 2m 57s trunk passed
          +1 compile 4m 13s trunk passed with JDK v1.8.0_60
          +1 compile 4m 7s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 56s trunk passed
          +1 mvneclipse 0m 53s trunk passed
          -1 findbugs 1m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 2m 15s trunk passed with JDK v1.8.0_60
          +1 javadoc 4m 55s trunk passed with JDK v1.7.0_79
          +1 mvninstall 3m 9s the patch passed
          +1 compile 4m 34s the patch passed with JDK v1.8.0_60
          +1 javac 4m 34s the patch passed
          +1 compile 4m 11s the patch passed with JDK v1.7.0_79
          +1 javac 4m 11s the patch passed
          -1 checkstyle 0m 58s Patch generated 4 new checkstyle issues in root (total was 453, now 454).
          +1 mvneclipse 0m 52s the patch passed
          -1 whitespace 0m 1s The patch has 2 line(s) with tabs.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          -1 findbugs 1m 6s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager introduced 1 new FindBugs issues.
          +1 javadoc 2m 12s the patch passed with JDK v1.8.0_60
          +1 javadoc 4m 49s the patch passed with JDK v1.7.0_79
          +1 unit 6m 46s hadoop-common in the patch passed with JDK v1.8.0_60.
          +1 unit 0m 21s hadoop-yarn-api in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 47s hadoop-yarn-common in the patch passed with JDK v1.8.0_60.
          +1 unit 8m 26s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_60.
          +1 unit 7m 7s hadoop-common in the patch passed with JDK v1.7.0_79.
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_79.
          +1 unit 2m 2s hadoop-yarn-common in the patch passed with JDK v1.7.0_79.
          +1 unit 9m 21s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          90m 4s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
            Possible doublecheck on org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.errorFileNamePattern in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorFileNamePattern(Configuration) At ContainerLaunch.java:org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorFileNamePattern(Configuration) At ContainerLaunch.java:[lines 429-431]



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-09
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771333/YARN-2934.v1.005.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml
          uname Linux e1f2401133ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / 668e897
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9643/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9643/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9643/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9643/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9643/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9643/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 2m 57s trunk passed +1 compile 4m 13s trunk passed with JDK v1.8.0_60 +1 compile 4m 7s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 56s trunk passed +1 mvneclipse 0m 53s trunk passed -1 findbugs 1m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 2m 15s trunk passed with JDK v1.8.0_60 +1 javadoc 4m 55s trunk passed with JDK v1.7.0_79 +1 mvninstall 3m 9s the patch passed +1 compile 4m 34s the patch passed with JDK v1.8.0_60 +1 javac 4m 34s the patch passed +1 compile 4m 11s the patch passed with JDK v1.7.0_79 +1 javac 4m 11s the patch passed -1 checkstyle 0m 58s Patch generated 4 new checkstyle issues in root (total was 453, now 454). +1 mvneclipse 0m 52s the patch passed -1 whitespace 0m 1s The patch has 2 line(s) with tabs. +1 xml 0m 0s The patch has no ill-formed XML file. -1 findbugs 1m 6s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager introduced 1 new FindBugs issues. +1 javadoc 2m 12s the patch passed with JDK v1.8.0_60 +1 javadoc 4m 49s the patch passed with JDK v1.7.0_79 +1 unit 6m 46s hadoop-common in the patch passed with JDK v1.8.0_60. +1 unit 0m 21s hadoop-yarn-api in the patch passed with JDK v1.8.0_60. +1 unit 1m 47s hadoop-yarn-common in the patch passed with JDK v1.8.0_60. +1 unit 8m 26s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_60. +1 unit 7m 7s hadoop-common in the patch passed with JDK v1.7.0_79. +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_79. +1 unit 2m 2s hadoop-yarn-common in the patch passed with JDK v1.7.0_79. +1 unit 9m 21s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 90m 4s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   Possible doublecheck on org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.errorFileNamePattern in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorFileNamePattern(Configuration) At ContainerLaunch.java:org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorFileNamePattern(Configuration) At ContainerLaunch.java: [lines 429-431] Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-09 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771333/YARN-2934.v1.005.patch JIRA Issue YARN-2934 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml uname Linux e1f2401133ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / 668e897 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9643/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9643/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9643/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9643/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9643/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9643/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Varun Saxena, Varun Vasudev, Rohith Sharma K S & others in the watchers list
          Can you guys take a look on the latest patch ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Varun Saxena , Varun Vasudev , Rohith Sharma K S & others in the watchers list Can you guys take a look on the latest patch ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi [~Varun Vasude], Rohith Sharma K S, Gera Shegalov, Bikas Saha and others, appreciate if any one can review this jira as its ready for review since a while !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi [~Varun Vasude] , Rohith Sharma K S , Gera Shegalov , Bikas Saha and others, appreciate if any one can review this jira as its ready for review since a while !
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks Naganarasimha G R! I skimmed the patch, it is in a pretty good shape. Aiming to give you more detailed feedback over next few days.

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks Naganarasimha G R ! I skimmed the patch, it is in a pretty good shape. Aiming to give you more detailed feedback over next few days.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Gera Shegalov !

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Gera Shegalov !
          Hide
          vvasudev Varun Vasudev added a comment -

          Naganarasimha G R - one minor comment - can we make the tail size a config parameter? Currently it's hard-coded in the code.

          Show
          vvasudev Varun Vasudev added a comment - Naganarasimha G R - one minor comment - can we make the tail size a config parameter? Currently it's hard-coded in the code.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Varun Vasudev,
          Hi as per the comment from Steve Loughran, he was of the opinion that it might be less chances that some one would configure it, but i am open to add additional configuration if not an issue !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Varun Vasudev , Hi as per the comment from Steve Loughran , he was of the opinion that it might be less chances that some one would configure it, but i am open to add additional configuration if not an issue !
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Varun Vasudev,
          Have created a new patch with configuration for the tail size, please review

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Varun Vasudev , Have created a new patch with configuration for the tail size, please review
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 6s trunk passed
          +1 compile 8m 47s trunk passed with JDK v1.8.0_66
          +1 compile 9m 31s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 2m 43s trunk passed
          +1 mvneclipse 0m 55s trunk passed
          +1 findbugs 5m 38s trunk passed
          +1 javadoc 2m 26s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 55s trunk passed with JDK v1.7.0_91
          +1 mvninstall 3m 10s the patch passed
          +1 compile 8m 49s the patch passed with JDK v1.8.0_66
          -1 javac 20m 18s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new issues (was 734, now 734).
          +1 javac 8m 49s the patch passed
          +1 compile 9m 29s the patch passed with JDK v1.7.0_91
          -1 javac 29m 47s root-jdk1.7.0_91 with JDK v1.7.0_91 generated 5 new issues (was 728, now 728).
          +1 javac 9m 29s the patch passed
          -1 checkstyle 1m 2s Patch generated 7 new checkstyle issues in root (total was 455, now 459).
          +1 mvnsite 2m 42s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) with tabs.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          -1 findbugs 1m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager introduced 1 new FindBugs issues.
          +1 javadoc 2m 26s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 57s the patch passed with JDK v1.7.0_91
          -1 unit 7m 38s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 8s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 12s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          -1 unit 7m 17s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 15s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 22s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          124m 48s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
            Possible doublecheck on org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.errorFileNamePattern in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorFileNamePattern(Configuration) At ContainerLaunch.java:org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorFileNamePattern(Configuration) At ContainerLaunch.java:[lines 430-432]
          JDK v1.8.0_66 Failed junit tests hadoop.ha.TestZKFailoverController
          JDK v1.7.0_91 Failed junit tests hadoop.ipc.TestIPC



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777354/YARN-2934.v1.006.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4c44af69aff0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7fb212e
          findbugs v3.0.0
          javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt
          javac root-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9962/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9962/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 6s trunk passed +1 compile 8m 47s trunk passed with JDK v1.8.0_66 +1 compile 9m 31s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 3s trunk passed +1 mvnsite 2m 43s trunk passed +1 mvneclipse 0m 55s trunk passed +1 findbugs 5m 38s trunk passed +1 javadoc 2m 26s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 55s trunk passed with JDK v1.7.0_91 +1 mvninstall 3m 10s the patch passed +1 compile 8m 49s the patch passed with JDK v1.8.0_66 -1 javac 20m 18s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new issues (was 734, now 734). +1 javac 8m 49s the patch passed +1 compile 9m 29s the patch passed with JDK v1.7.0_91 -1 javac 29m 47s root-jdk1.7.0_91 with JDK v1.7.0_91 generated 5 new issues (was 728, now 728). +1 javac 9m 29s the patch passed -1 checkstyle 1m 2s Patch generated 7 new checkstyle issues in root (total was 455, now 459). +1 mvnsite 2m 42s the patch passed +1 mvneclipse 0m 55s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) with tabs. +1 xml 0m 0s The patch has no ill-formed XML file. -1 findbugs 1m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager introduced 1 new FindBugs issues. +1 javadoc 2m 26s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 57s the patch passed with JDK v1.7.0_91 -1 unit 7m 38s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 8s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 9m 12s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. -1 unit 7m 17s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 15s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 9m 22s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 124m 48s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   Possible doublecheck on org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.errorFileNamePattern in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorFileNamePattern(Configuration) At ContainerLaunch.java:org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.getErrorFileNamePattern(Configuration) At ContainerLaunch.java: [lines 430-432] JDK v1.8.0_66 Failed junit tests hadoop.ha.TestZKFailoverController JDK v1.7.0_91 Failed junit tests hadoop.ipc.TestIPC Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777354/YARN-2934.v1.006.patch JIRA Issue YARN-2934 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4c44af69aff0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7fb212e findbugs v3.0.0 javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt javac root-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9962/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9962/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9962/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Naganarasimha G R,
          Please make sure that the patch does not introduce new problems. Both checkstyle and findbugs report problems related to the patch. Check the Hadoop QA comment above. Keep addressing the newly introduced issues without waiting for review to simplify the review process.

          I suggest to use globs instead of regexes, so you can simply call FileSystem#globStatus. The path pattern could be something like

          {*stderr*,*STDERR*}

          or maybe

          {*err,*ERR,*out,*OUT}

          . I'd rather have a longer config value than adding more code to make patterns case-insensitive. In practice we mostly need stderr

          Not sure how fancy we need to be with the case where multiple log files qualify for the pattern, but maybe at least mention to the user there are more files to look at.

          In general, don't try optimize for the failure case. Things like

          private static long tailSizeInBytes = -1;
          

          look like a bug. Simply get it from conf exactly when it's needed.

          Show
          jira.shegalov Gera Shegalov added a comment - Hi Naganarasimha G R , Please make sure that the patch does not introduce new problems. Both checkstyle and findbugs report problems related to the patch. Check the Hadoop QA comment above. Keep addressing the newly introduced issues without waiting for review to simplify the review process. I suggest to use globs instead of regexes, so you can simply call FileSystem#globStatus. The path pattern could be something like {*stderr*,*STDERR*} or maybe {*err,*ERR,*out,*OUT} . I'd rather have a longer config value than adding more code to make patterns case-insensitive. In practice we mostly need stderr Not sure how fancy we need to be with the case where multiple log files qualify for the pattern, but maybe at least mention to the user there are more files to look at. In general, don't try optimize for the failure case. Things like private static long tailSizeInBytes = -1; look like a bug. Simply get it from conf exactly when it's needed.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the review Gera Shegalov,

          Please make sure that the patch does not introduce new problems. Both checkstyle and findbugs report problems related to the patch.

          Well, was earlier not sure whether the findbugs issue reported was valid one, went through the description given in the finds bugs report got the understanding about it! Thought of correcting the synchronization issue but based on your last comment made it to fetch everytime from the conf, in the latest patch.
          Most of the issues reported in the checkstyle are not directly induced by the patch or is ok to live with. Valid issues are incorporated in the latest patch

          I'd rather have a longer config value than adding more code to make patterns case-insensitive. In practice we mostly need stderr

          Would like to differ here, IMHO code added is not much and anyway i have already finished coding it and when there is a way possible to avoid configuring multiple cases, then why expect the user to configure for both cases ? Though most cases stderr is sufficient, its like any kind of app can be submitted and the pattern can be mix and match of the cases too, ex . in distributed shell client for am log "AppMaster.stderr" is used as the error file name (emphasizing different cases here though it has stderr).
          I am open to modify to glob approach if there is any flaw/disadvantage in the current approach !

          In general, don't try optimize for the failure case. Things like look like a bug. Simply get it from conf exactly when it's needed.

          Ok have corrected for the tail size and in the similar lines for pattern too, in the latest patch.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review Gera Shegalov , Please make sure that the patch does not introduce new problems. Both checkstyle and findbugs report problems related to the patch. Well, was earlier not sure whether the findbugs issue reported was valid one, went through the description given in the finds bugs report got the understanding about it! Thought of correcting the synchronization issue but based on your last comment made it to fetch everytime from the conf, in the latest patch. Most of the issues reported in the checkstyle are not directly induced by the patch or is ok to live with. Valid issues are incorporated in the latest patch I'd rather have a longer config value than adding more code to make patterns case-insensitive. In practice we mostly need stderr Would like to differ here, IMHO code added is not much and anyway i have already finished coding it and when there is a way possible to avoid configuring multiple cases, then why expect the user to configure for both cases ? Though most cases stderr is sufficient, its like any kind of app can be submitted and the pattern can be mix and match of the cases too, ex . in distributed shell client for am log "AppMaster.stderr" is used as the error file name (emphasizing different cases here though it has stderr). I am open to modify to glob approach if there is any flaw/disadvantage in the current approach ! In general, don't try optimize for the failure case. Things like look like a bug. Simply get it from conf exactly when it's needed. Ok have corrected for the tail size and in the similar lines for pattern too, in the latest patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 10m 59s trunk passed
          +1 compile 17m 0s trunk passed with JDK v1.8.0_66
          +1 compile 13m 42s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 22s trunk passed
          +1 mvnsite 3m 37s trunk passed
          +1 mvneclipse 1m 20s trunk passed
          +1 findbugs 7m 22s trunk passed
          +1 javadoc 3m 41s trunk passed with JDK v1.8.0_66
          +1 javadoc 6m 4s trunk passed with JDK v1.7.0_91
          +1 mvninstall 4m 19s the patch passed
          +1 compile 13m 23s the patch passed with JDK v1.8.0_66
          +1 javac 13m 23s the patch passed
          +1 compile 12m 8s the patch passed with JDK v1.7.0_91
          +1 javac 12m 8s the patch passed
          -1 checkstyle 1m 19s Patch generated 3 new checkstyle issues in root (total was 455, now 455).
          +1 mvnsite 3m 33s the patch passed
          +1 mvneclipse 1m 17s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) with tabs.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 8m 6s the patch passed
          +1 javadoc 3m 38s the patch passed with JDK v1.8.0_66
          +1 javadoc 6m 10s the patch passed with JDK v1.7.0_91
          -1 unit 9m 57s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 40s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 50s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 10m 38s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66.
          +1 unit 9m 38s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 0m 37s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 44s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 10m 21s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 33s Patch does not generate ASF License warnings.
          169m 26s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestRPC
            hadoop.fs.shell.find.TestFind
            hadoop.test.TestTimedOutTestsListener
            hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778177/YARN-2934.v1.007.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 38b7ab886fe4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f741476
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10015/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10015/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 59s trunk passed +1 compile 17m 0s trunk passed with JDK v1.8.0_66 +1 compile 13m 42s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 22s trunk passed +1 mvnsite 3m 37s trunk passed +1 mvneclipse 1m 20s trunk passed +1 findbugs 7m 22s trunk passed +1 javadoc 3m 41s trunk passed with JDK v1.8.0_66 +1 javadoc 6m 4s trunk passed with JDK v1.7.0_91 +1 mvninstall 4m 19s the patch passed +1 compile 13m 23s the patch passed with JDK v1.8.0_66 +1 javac 13m 23s the patch passed +1 compile 12m 8s the patch passed with JDK v1.7.0_91 +1 javac 12m 8s the patch passed -1 checkstyle 1m 19s Patch generated 3 new checkstyle issues in root (total was 455, now 455). +1 mvnsite 3m 33s the patch passed +1 mvneclipse 1m 17s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) with tabs. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 8m 6s the patch passed +1 javadoc 3m 38s the patch passed with JDK v1.8.0_66 +1 javadoc 6m 10s the patch passed with JDK v1.7.0_91 -1 unit 9m 57s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 40s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 50s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 10m 38s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66. +1 unit 9m 38s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 37s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 44s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 10m 21s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 33s Patch does not generate ASF License warnings. 169m 26s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestRPC   hadoop.fs.shell.find.TestFind   hadoop.test.TestTimedOutTestsListener   hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778177/YARN-2934.v1.007.patch JIRA Issue YARN-2934 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 38b7ab886fe4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f741476 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10015/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10015/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10015/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          -1 on manual regexes in favor of code reuse. 99.9% of YARN users will never change this conf. Simple globs I was suggesting already cover even AppMaster.stderr.

          In ContainerLaunch#getErrorLogTail
          get rid of

              if (containerLogDir == null) {
                return null;
              }
          

          it cannot be null

          Show
          jira.shegalov Gera Shegalov added a comment - -1 on manual regexes in favor of code reuse. 99.9% of YARN users will never change this conf. Simple globs I was suggesting already cover even AppMaster.stderr. In ContainerLaunch#getErrorLogTail get rid of if (containerLogDir == null ) { return null ; } it cannot be null
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Simple globs I was suggesting already cover even AppMaster.stderr.

          Yep i agree i too mentioned the same in my earlier comment to emphasizing different cases here though it has stderr , what i was trying to point there was app developers can give a name of file in mixed cases (like *stdErr,*sysLog) and it cant be enforced by yarn. so it would be difficult for admin to determine and give for all kinds of app being run.

          99.9% of YARN users will never change this conf.

          Definitely this feature is so rare that admin is going to configure repeatedly but my intention is to just make admin's work lil more easy.

          If you still weigh code resuse as more important than to make admin config multiple patterns of the log filename then i will update the patch accordingly, waiting for your feedback.

          In ContainerLaunch#getErrorLogTail get rid of .. null check ...

          Agree, will correct in next patch long with the test case failure

          Show
          Naganarasimha Naganarasimha G R added a comment - Simple globs I was suggesting already cover even AppMaster.stderr. Yep i agree i too mentioned the same in my earlier comment to emphasizing different cases here though it has stderr , what i was trying to point there was app developers can give a name of file in mixed cases (like *stdErr,*sysLog) and it cant be enforced by yarn. so it would be difficult for admin to determine and give for all kinds of app being run. 99.9% of YARN users will never change this conf. Definitely this feature is so rare that admin is going to configure repeatedly but my intention is to just make admin's work lil more easy. If you still weigh code resuse as more important than to make admin config multiple patterns of the log filename then i will update the patch accordingly, waiting for your feedback. In ContainerLaunch#getErrorLogTail get rid of .. null check ... Agree, will correct in next patch long with the test case failure
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Uploading a patch with correcting testcase, whitespace and some review comments.
          Varun Vasudev, Rohith Sharma K S,
          Would like to get your views on using globs or regex, based on that can upload a final patch !

          Show
          Naganarasimha Naganarasimha G R added a comment - Uploading a patch with correcting testcase, whitespace and some review comments. Varun Vasudev , Rohith Sharma K S , Would like to get your views on using globs or regex , based on that can upload a final patch !
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Use RawLocalFileSystem, we don't need the checksumming version: FileSystem fileSystem = FileSystem.getLocal(conf).getRaw()

          Show
          jira.shegalov Gera Shegalov added a comment - Use RawLocalFileSystem, we don't need the checksumming version: FileSystem fileSystem = FileSystem.getLocal(conf).getRaw()
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          oops missed this comment had uploaded the latest patch at the same time ... will take care of this as part of the next patch...

          Show
          Naganarasimha Naganarasimha G R added a comment - oops missed this comment had uploaded the latest patch at the same time ... will take care of this as part of the next patch...
          Hide
          jira.shegalov Gera Shegalov added a comment -

          That should go to the exception message

          422      } else if (listStatus.length > 1) {
          423	      LOG.warn("Multiple files in " + containerLogDir
          424	          + ", seems to match the error file name pattern configured ");
          425      }
          

          Don't do branching, pass the string builder diagnosticInfo and do something like

          diagnosticInfo
                        .append("Error files: ")
                        .append(Arrays.toString(listStatus))
                        .append("\n")
                        .append("Last ").append(tailBytes).append(" bytes of ").append(listStatus[0])
                        .append(new String(tailBytes, UTF_8));
          
          Show
          jira.shegalov Gera Shegalov added a comment - That should go to the exception message 422 } else if (listStatus.length > 1) { 423 LOG.warn( "Multiple files in " + containerLogDir 424 + ", seems to match the error file name pattern configured " ); 425 } Don't do branching, pass the string builder diagnosticInfo and do something like diagnosticInfo .append( "Error files: " ) .append(Arrays.toString(listStatus)) .append( "\n" ) .append( "Last " ).append(tailBytes).append( " bytes of " ).append(listStatus[0]) .append( new String (tailBytes, UTF_8));
          Hide
          vvasudev Varun Vasudev added a comment -

          I'm not too fussed either way. Globs are probably easier for an end user. Seeing as Gera Shegalov feels strongly about this, I'll defer to him unless you have an example app log file that a glob can't catch but a regex would.

          Show
          vvasudev Varun Vasudev added a comment - I'm not too fussed either way. Globs are probably easier for an end user. Seeing as Gera Shegalov feels strongly about this, I'll defer to him unless you have an example app log file that a glob can't catch but a regex would.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 1s trunk passed
          +1 compile 8m 43s trunk passed with JDK v1.8.0_66
          +1 compile 10m 1s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 13s trunk passed
          +1 mvnsite 2m 51s trunk passed
          +1 mvneclipse 0m 54s trunk passed
          +1 findbugs 6m 41s trunk passed
          +1 javadoc 2m 29s trunk passed with JDK v1.8.0_66
          +1 javadoc 5m 2s trunk passed with JDK v1.7.0_91
          +1 mvninstall 3m 6s the patch passed
          +1 compile 8m 40s the patch passed with JDK v1.8.0_66
          +1 javac 8m 40s the patch passed
          +1 compile 9m 22s the patch passed with JDK v1.7.0_91
          +1 javac 9m 22s the patch passed
          -1 checkstyle 1m 2s Patch generated 3 new checkstyle issues in root (total was 455, now 455).
          +1 mvnsite 2m 38s the patch passed
          +1 mvneclipse 0m 54s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 6m 13s the patch passed
          +1 javadoc 2m 26s the patch passed with JDK v1.8.0_66
          +1 javadoc 5m 2s the patch passed with JDK v1.7.0_91
          +1 unit 8m 15s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 59s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 43s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          -1 unit 7m 58s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 12s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 14s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          126m 38s



          Reason Tests
          JDK v1.7.0_91 Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778228/YARN-2934.v1.008.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 861898cc9a41 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d90625e
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10020/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10020/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10020/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10020/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10020/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 1s trunk passed +1 compile 8m 43s trunk passed with JDK v1.8.0_66 +1 compile 10m 1s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 13s trunk passed +1 mvnsite 2m 51s trunk passed +1 mvneclipse 0m 54s trunk passed +1 findbugs 6m 41s trunk passed +1 javadoc 2m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 5m 2s trunk passed with JDK v1.7.0_91 +1 mvninstall 3m 6s the patch passed +1 compile 8m 40s the patch passed with JDK v1.8.0_66 +1 javac 8m 40s the patch passed +1 compile 9m 22s the patch passed with JDK v1.7.0_91 +1 javac 9m 22s the patch passed -1 checkstyle 1m 2s Patch generated 3 new checkstyle issues in root (total was 455, now 455). +1 mvnsite 2m 38s the patch passed +1 mvneclipse 0m 54s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 6m 13s the patch passed +1 javadoc 2m 26s the patch passed with JDK v1.8.0_66 +1 javadoc 5m 2s the patch passed with JDK v1.7.0_91 +1 unit 8m 15s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 59s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 43s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. -1 unit 7m 58s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 12s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 9m 14s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 126m 38s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778228/YARN-2934.v1.008.patch JIRA Issue YARN-2934 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 861898cc9a41 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d90625e findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10020/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10020/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10020/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10020/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10020/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the review comments and opinions Gera Shegalov & Varun Vasudev,

          Seeing as Gera Shegalov feels strongly about this, I'll defer to him unless you have an example app log file that a glob can't catch but a regex would.

          Well i don't have a strong case to back my hunch and neither could i get a better example to show advantages of regex over glob. Most of the cases(MR, Distributed shell, Spark etc...) are all using stderr, so i am attaching a patch with modifications incorporating Gera Shegalov's suggestion along with fixes for other issues.

          diagnosticInfo
                        .append("Error files: ")
                        .append(Arrays.toString(listStatus))
                        .append("\n")
                        .append("Last ").append(tailBytes).append(" bytes of ").append(listStatus[0])
                        .append(new String(tailBytes, UTF_8));
          

          was providing a result as

          Container exited with a non-zero exit code 127. Error file(s): [DeprecatedRawLocalFileStatus{path=file:/opt/git/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/TestContainerLaunch-localLogDir/application_1450367986801_0001/container_1450367986801_0001_01_000001/error.log; isDirectory=false; length=60; replication=1; blocksize=33554432; modification_time=1450367987000; access_time=1450367987000; owner=; group=; permission=rw-rw-rw-; isSymlink=false}]
          Last [B@78a7f540 bytes of DeprecatedRawLocalFileStatus{path=file:/opt/git/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/TestContainerLaunch-localLogDir/application_1450367986801_0001/container_1450367986801_0001_01_000001/error.log; isDirectory=false; length=60; replication=1; blocksize=33554432; modification_time=1450367987000; access_time=1450367987000; owner=; group=; permission=rw-rw-rw-; isSymlink=false} :
          /bin/bash: /no/jvm/here/bin/java: No such file or directory
          

          so have modified it to be

                    diagnosticInfo
                        .append("Last ").append(tailSizeInBytes).append(" bytes of ")
                        .append(listStatus[0].getPath().getName())
                        .append(" :\n")
                        .append(new String(tailBytes, UTF_8));
          

          Hope its fine else can revert it as per your suggestion.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review comments and opinions Gera Shegalov & Varun Vasudev , Seeing as Gera Shegalov feels strongly about this, I'll defer to him unless you have an example app log file that a glob can't catch but a regex would. Well i don't have a strong case to back my hunch and neither could i get a better example to show advantages of regex over glob. Most of the cases(MR, Distributed shell, Spark etc...) are all using stderr , so i am attaching a patch with modifications incorporating Gera Shegalov 's suggestion along with fixes for other issues. diagnosticInfo .append( "Error files: " ) .append(Arrays.toString(listStatus)) .append( "\n" ) .append( "Last " ).append(tailBytes).append( " bytes of " ).append(listStatus[0]) .append( new String (tailBytes, UTF_8)); was providing a result as Container exited with a non-zero exit code 127. Error file(s): [DeprecatedRawLocalFileStatus{path=file:/opt/git/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/TestContainerLaunch-localLogDir/application_1450367986801_0001/container_1450367986801_0001_01_000001/error.log; isDirectory= false ; length=60; replication=1; blocksize=33554432; modification_time=1450367987000; access_time=1450367987000; owner=; group=; permission=rw-rw-rw-; isSymlink= false }] Last [B@78a7f540 bytes of DeprecatedRawLocalFileStatus{path=file:/opt/git/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/TestContainerLaunch-localLogDir/application_1450367986801_0001/container_1450367986801_0001_01_000001/error.log; isDirectory= false ; length=60; replication=1; blocksize=33554432; modification_time=1450367987000; access_time=1450367987000; owner=; group=; permission=rw-rw-rw-; isSymlink= false } : /bin/bash: /no/jvm/here/bin/java: No such file or directory so have modified it to be diagnosticInfo .append( "Last " ).append(tailSizeInBytes).append( " bytes of " ) .append(listStatus[0].getPath().getName()) .append( " :\n" ) .append( new String (tailBytes, UTF_8)); Hope its fine else can revert it as per your suggestion.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 1s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 55s trunk passed
          +1 compile 1m 50s trunk passed with JDK v1.8.0_66
          +1 compile 2m 8s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 1m 29s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 3m 30s trunk passed
          +1 javadoc 1m 26s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 45s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 21s the patch passed
          +1 compile 1m 52s the patch passed with JDK v1.8.0_66
          +1 javac 1m 52s the patch passed
          +1 compile 2m 8s the patch passed with JDK v1.7.0_91
          +1 javac 2m 8s the patch passed
          -1 checkstyle 0m 27s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 279, now 280).
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 3m 54s the patch passed
          +1 javadoc 1m 24s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 48s the patch passed with JDK v1.7.0_91
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 54s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 38s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 7s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          64m 30s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778278/YARN-2934.v2.001.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 1ea9566ad72c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8d278d8
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10022/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10022/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10022/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 55s trunk passed +1 compile 1m 50s trunk passed with JDK v1.8.0_66 +1 compile 2m 8s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 29s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 3m 30s trunk passed +1 javadoc 1m 26s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 45s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 21s the patch passed +1 compile 1m 52s the patch passed with JDK v1.8.0_66 +1 javac 1m 52s the patch passed +1 compile 2m 8s the patch passed with JDK v1.7.0_91 +1 javac 2m 8s the patch passed -1 checkstyle 0m 27s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 279, now 280). +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 54s the patch passed +1 javadoc 1m 24s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 48s the patch passed with JDK v1.7.0_91 +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 54s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 38s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 9m 7s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 64m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778278/YARN-2934.v2.001.patch JIRA Issue YARN-2934 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1ea9566ad72c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8d278d8 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10022/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10022/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10022/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          checkstyle is not valid or not induced due to this patch !

          Show
          Naganarasimha Naganarasimha G R added a comment - checkstyle is not valid or not induced due to this patch !
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks for the latest patch. good to see the patch lose 3kb, most of all there are no more changes to the common Configuration class.

          One checkstyle issue is the 80-column warning is from the patch around:

                  long tailSizeInBytes = conf.getLong(
                      YarnConfiguration.NM_CONTAINER_ERROR_FILE_TAIL_SIZE_IN_BYTES,
                      YarnConfiguration.DEFAULT_NM_CONTAINER_ERROR_FILE_TAIL_SIZE_IN_BYTES);
          

          Those are pretty long names:
          Can we do:
          container.stderr.tail.bytes
          NM_CONTAINER_STDERR_BYTES
          and the corresponding default. Having stderr in name is also great for users to understand what error file is meant in 99% of the cases.
          Same thing is for container.stderr.pattern

          Still don't see any value in this, please drop:

                  if (listStatus.length > 1) {
                    LOG.error("Multiple files in " + containerLogDir
                        + ", seems to match the error file name pattern configured. "
                        + Arrays.toString(listStatus));
                  }
          

          Let us not guard the tail read:

                  if (fileSize != 0) {
          

          and there is a value in seeing that the file is empty already on the client-side.

          Instead of

          IOUtils.closeStream(errorFileIS) 
          

          call cleanup so we can pass the logger

          IOUtils.cleanup(LOG, errorFileIS)
          

          Since the trunk is on JDK7 min:
          We can drop the constant UTF_8 and use
          in

          new String(tailBytes, StandardCharsets.UTF_8)
          

          listStatus as a name for a variable is not intuitive. Maybe use errFileStatus for that.

          Obviously I meant tailSizeInBytes, thanks for paying attention. Agree that RLFS file status toString might look too ugly.

          We can FileUtil.stat2Paths or add a loop here to extract just the last path component.

          Also realizing that we should have a low cap on the tail size to prevent a misconfiguration to knock out NM with OOM on container failures since we do:

          byte[] tailBytes = new byte[bufferSize];
          

          One can easily see why I initially confused tailBytes for an int. It should be called along the lines

          tailBuffer

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks for the latest patch. good to see the patch lose 3kb, most of all there are no more changes to the common Configuration class. One checkstyle issue is the 80-column warning is from the patch around: long tailSizeInBytes = conf.getLong( YarnConfiguration.NM_CONTAINER_ERROR_FILE_TAIL_SIZE_IN_BYTES, YarnConfiguration.DEFAULT_NM_CONTAINER_ERROR_FILE_TAIL_SIZE_IN_BYTES); Those are pretty long names: Can we do: container.stderr.tail.bytes NM_CONTAINER_STDERR_BYTES and the corresponding default. Having stderr in name is also great for users to understand what error file is meant in 99% of the cases. Same thing is for container.stderr.pattern Still don't see any value in this, please drop: if (listStatus.length > 1) { LOG.error( "Multiple files in " + containerLogDir + ", seems to match the error file name pattern configured. " + Arrays.toString(listStatus)); } Let us not guard the tail read: if (fileSize != 0) { and there is a value in seeing that the file is empty already on the client-side. Instead of IOUtils.closeStream(errorFileIS) call cleanup so we can pass the logger IOUtils.cleanup(LOG, errorFileIS) Since the trunk is on JDK7 min: We can drop the constant UTF_8 and use in new String (tailBytes, StandardCharsets.UTF_8) listStatus as a name for a variable is not intuitive. Maybe use errFileStatus for that. Obviously I meant tailSizeInBytes, thanks for paying attention. Agree that RLFS file status toString might look too ugly. We can FileUtil.stat2Paths or add a loop here to extract just the last path component. Also realizing that we should have a low cap on the tail size to prevent a misconfiguration to knock out NM with OOM on container failures since we do: byte [] tailBytes = new byte [bufferSize]; One can easily see why I initially confused tailBytes for an int. It should be called along the lines tailBuffer
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Gera Shegalov thanks for the comments,

          good to see the patch lose 3kb, most of all there are no more changes to the common Configuration class.

          Still don't see any value in this, please drop: if (listStatus.length > 1) ....

          I placed it as i understood your comment Not sure how fancy we need to be with the case where multiple log files qualify for the pattern, but maybe at least mention to the user there are more files to look at. in this way. But felt its better to have a log than showing it in the diagnostic msg , thoughts ?

          We can FileUtil.stat2Paths or add a loop here to extract just the last path component.

          Going by the propability, in most of the cases there will be only one error file matching the pattern, in that case based on your earlier approach which you had suggested will repeat the name of the file twice, which i felt redundant hence removed it. Message would look something like :

          Container exited with a non-zero exit code 127. Error file(s): [error.log]
          Last 4096 bytes of error.log :
          /bin/bash: /no/jvm/here/bin/java: No such file or directory
          

          Please share your thoughts if required can add it and for other comments i am working on it .

          Show
          Naganarasimha Naganarasimha G R added a comment - Gera Shegalov thanks for the comments, good to see the patch lose 3kb, most of all there are no more changes to the common Configuration class. Still don't see any value in this, please drop: if (listStatus.length > 1) .... I placed it as i understood your comment Not sure how fancy we need to be with the case where multiple log files qualify for the pattern, but maybe at least mention to the user there are more files to look at. in this way. But felt its better to have a log than showing it in the diagnostic msg , thoughts ? We can FileUtil.stat2Paths or add a loop here to extract just the last path component. Going by the propability, in most of the cases there will be only one error file matching the pattern, in that case based on your earlier approach which you had suggested will repeat the name of the file twice, which i felt redundant hence removed it. Message would look something like : Container exited with a non-zero exit code 127. Error file(s): [error.log] Last 4096 bytes of error.log : /bin/bash: /no/jvm/here/bin/java: No such file or directory Please share your thoughts if required can add it and for other comments i am working on it .
          Hide
          jira.shegalov Gera Shegalov added a comment -

          the message looks good to me.

          Show
          jira.shegalov Gera Shegalov added a comment - the message looks good to me.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Regarding my comment about the user, I meant the YARN app user. The users don't look at the NM logs. They look at the exceptions in the webUI and on the client side. if the exception says

          Container exited with a non-zero exit code 127. Error file(s): [error.log, stderr.1, stderr.2]
          Last 4096 bytes of error.log :
          /bin/bash: /no/jvm/here/bin/java: No such file or directory
          

          the user will know it should also check those.

          Show
          jira.shegalov Gera Shegalov added a comment - Regarding my comment about the user, I meant the YARN app user. The users don't look at the NM logs. They look at the exceptions in the webUI and on the client side. if the exception says Container exited with a non-zero exit code 127. Error file(s): [error.log, stderr.1, stderr.2] Last 4096 bytes of error.log : /bin/bash: /no/jvm/here/bin/java: No such file or directory the user will know it should also check those.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Gera Shegalov,
          Ok got your intention, Just one suggestion shall i add {{Error file(s): [error.log, stderr.1, stderr.2] only when there are multiple files ? if not file name just gets repeated twice ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Gera Shegalov , Ok got your intention, Just one suggestion shall i add {{Error file(s): [error.log, stderr.1, stderr.2] only when there are multiple files ? if not file name just gets repeated twice ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Gera Shegalov,
          Have incorporated all the comments in the latest patch, can you please take a look !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Gera Shegalov , Have incorporated all the comments in the latest patch, can you please take a look !
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Minor repetition is not big of a deal, IMO.

          The reason I thought of printing file statuses is that you see the file size. Which brings us to the following point in the fanciness area. Right now we are blindly grabbing file 0. It would however make much more sense to grab the most recent (highest mtime) non-empty file.

          Show
          jira.shegalov Gera Shegalov added a comment - Minor repetition is not big of a deal, IMO. The reason I thought of printing file statuses is that you see the file size. Which brings us to the following point in the fanciness area. Right now we are blindly grabbing file 0. It would however make much more sense to grab the most recent (highest mtime) non-empty file.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Minor repetition is not big of a deal, IMO.

          The reason I thought of printing file statuses is that you see the file size. Which brings us to the following point in the fanciness area. Right now we are blindly grabbing file 0. It would however make much more sense to grab the most recent (highest mtime) non-empty file.

          Show
          jira.shegalov Gera Shegalov added a comment - Minor repetition is not big of a deal, IMO. The reason I thought of printing file statuses is that you see the file size. Which brings us to the following point in the fanciness area. Right now we are blindly grabbing file 0. It would however make much more sense to grab the most recent (highest mtime) non-empty file.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Gera Shegalov,

          Minor repetition is not big of a deal, IMO.

          could please have a look at the latest patch if fine then will not rework, else will try to update as per your comment

          Right now we are blindly grabbing file 0. It would however make much more sense to grab the most recent (highest mtime) non-empty file.

          Are you suggesting for this change in this patch or just mentioning that its good to have ? If required can rework as it would be better to go for the latest modified file !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Gera Shegalov , Minor repetition is not big of a deal, IMO. could please have a look at the latest patch if fine then will not rework, else will try to update as per your comment Right now we are blindly grabbing file 0. It would however make much more sense to grab the most recent (highest mtime) non-empty file. Are you suggesting for this change in this patch or just mentioning that its good to have ? If required can rework as it would be better to go for the latest modified file !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 23m 56s trunk passed
          +1 compile 10m 39s trunk passed with JDK v1.8.0_66
          +1 compile 7m 2s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 15s trunk passed
          +1 mvnsite 4m 26s trunk passed
          +1 mvneclipse 1m 32s trunk passed
          +1 findbugs 10m 16s trunk passed
          +1 javadoc 5m 40s trunk passed with JDK v1.8.0_66
          +1 javadoc 8m 28s trunk passed with JDK v1.7.0_91
          +1 mvninstall 3m 53s the patch passed
          +1 compile 9m 54s the patch passed with JDK v1.8.0_66
          +1 javac 9m 54s the patch passed
          +1 compile 6m 38s the patch passed with JDK v1.7.0_91
          +1 javac 6m 38s the patch passed
          -1 checkstyle 1m 12s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 279, now 279).
          +1 mvnsite 4m 20s the patch passed
          +1 mvneclipse 1m 33s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 11m 21s the patch passed
          +1 javadoc 5m 36s the patch passed with JDK v1.8.0_66
          +1 javadoc 7m 19s the patch passed with JDK v1.7.0_91
          +1 unit 1m 34s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 5m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 17m 27s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66.
          +1 unit 1m 14s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          -1 unit 5m 25s hadoop-yarn-common in the patch failed with JDK v1.7.0_91.
          -1 unit 15m 23s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91.
          +1 asflicense 1m 11s Patch does not generate ASF License warnings.
          178m 15s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.nodemanager.TestNodeManagerResync
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.util.resource.TestResources
            hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778417/YARN-2934.v2.002.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4ed2fb6dffee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 061c05c
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10033/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10033/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 23m 56s trunk passed +1 compile 10m 39s trunk passed with JDK v1.8.0_66 +1 compile 7m 2s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 15s trunk passed +1 mvnsite 4m 26s trunk passed +1 mvneclipse 1m 32s trunk passed +1 findbugs 10m 16s trunk passed +1 javadoc 5m 40s trunk passed with JDK v1.8.0_66 +1 javadoc 8m 28s trunk passed with JDK v1.7.0_91 +1 mvninstall 3m 53s the patch passed +1 compile 9m 54s the patch passed with JDK v1.8.0_66 +1 javac 9m 54s the patch passed +1 compile 6m 38s the patch passed with JDK v1.7.0_91 +1 javac 6m 38s the patch passed -1 checkstyle 1m 12s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 279, now 279). +1 mvnsite 4m 20s the patch passed +1 mvneclipse 1m 33s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 11m 21s the patch passed +1 javadoc 5m 36s the patch passed with JDK v1.8.0_66 +1 javadoc 7m 19s the patch passed with JDK v1.7.0_91 +1 unit 1m 34s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 5m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 17m 27s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66. +1 unit 1m 14s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. -1 unit 5m 25s hadoop-yarn-common in the patch failed with JDK v1.7.0_91. -1 unit 15m 23s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 1m 11s Patch does not generate ASF License warnings. 178m 15s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.nodemanager.TestNodeManagerResync JDK v1.7.0_91 Failed junit tests hadoop.yarn.util.resource.TestResources   hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778417/YARN-2934.v2.002.patch JIRA Issue YARN-2934 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4ed2fb6dffee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 061c05c findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/10033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10033/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10033/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Gera Shegalov,
          Incorporating changes for Right now we are blindly grabbing file 0. It would however make much more sense to grab the most recent (highest mtime) non-empty file,
          Please review & Earlier Test cases are not related to the modifications in this patch and its locally passing

          Show
          Naganarasimha Naganarasimha G R added a comment - Gera Shegalov , Incorporating changes for Right now we are blindly grabbing file 0. It would however make much more sense to grab the most recent (highest mtime) non-empty file , Please review & Earlier Test cases are not related to the modifications in this patch and its locally passing
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 30s trunk passed
          +1 compile 2m 15s trunk passed with JDK v1.8.0_66
          +1 compile 2m 21s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 38s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 53s trunk passed
          +1 javadoc 1m 34s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 0s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 28s the patch passed
          +1 compile 2m 6s the patch passed with JDK v1.8.0_66
          +1 javac 2m 6s the patch passed
          +1 compile 2m 21s the patch passed with JDK v1.7.0_91
          +1 javac 2m 21s the patch passed
          -1 checkstyle 0m 31s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 279, now 279).
          +1 mvnsite 1m 35s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 21s the patch passed
          +1 javadoc 1m 33s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 51s the patch passed with JDK v1.7.0_91
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 1s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 15s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 30s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          69m 26s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778536/YARN-2934.v2.003.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux dbffa7fbf0bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e63388f
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10043/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10043/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10043/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 30s trunk passed +1 compile 2m 15s trunk passed with JDK v1.8.0_66 +1 compile 2m 21s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 38s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 53s trunk passed +1 javadoc 1m 34s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 0s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 28s the patch passed +1 compile 2m 6s the patch passed with JDK v1.8.0_66 +1 javac 2m 6s the patch passed +1 compile 2m 21s the patch passed with JDK v1.7.0_91 +1 javac 2m 21s the patch passed -1 checkstyle 0m 31s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 279, now 279). +1 mvnsite 1m 35s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 21s the patch passed +1 javadoc 1m 33s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 51s the patch passed with JDK v1.7.0_91 +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 9m 1s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 15s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 9m 30s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 69m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778536/YARN-2934.v2.003.patch JIRA Issue YARN-2934 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux dbffa7fbf0bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e63388f findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10043/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10043/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10043/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Gera Shegalov, Hope latest patch covers all your comments, can you please take a look at it once ?
          check style issues are not directly induced by the patch.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Gera Shegalov , Hope latest patch covers all your comments, can you please take a look at it once ? check style issues are not directly induced by the patch.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Naganarasimha G R. Thanks for updating the patch.

          Things we have not addressed from my previous comments is capping the buffer size. But I now think it's good enough because we have a good small default for the tail NM_CONTAINER_STDERR_BYTES.

          Still please rename:

          -      FileStatus[] listStatus = fileSystem
          +      FileStatus[] errorStatuses = fileSystem
          

          or similar. It's an array of statuses and not status of a list

          Let us have a space after ',' and a new line in:

          -              .append(StringUtils.arrayToString(errorFileNames)).append(". ");
          +              .append(StringUtils.join(", ", errorFileNames)).append(".\n");
          

          Fix the test code accordingly

          method verifyTailErrorLogOnContainerExit can/should be private. Same for ContainerExitHandler class.

          Assume.assumeTrue(Shell.LINUX);
          should be
          Assume.assumeFalse(Shell.WINDOWS || Shell.OTHER);
          but actually why do we need this? The test seems to be platform-independent.

          Assert.assertNotNull(exitEvent.getDiagnosticInfo());

          seems redundant because you then have other asserts implying this already. I suggest to LOG.info the diagnostics instead to make the test log more useful.

          Show
          jira.shegalov Gera Shegalov added a comment - Hi Naganarasimha G R . Thanks for updating the patch. Things we have not addressed from my previous comments is capping the buffer size. But I now think it's good enough because we have a good small default for the tail NM_CONTAINER_STDERR_BYTES. Still please rename: - FileStatus[] listStatus = fileSystem + FileStatus[] errorStatuses = fileSystem or similar. It's an array of statuses and not status of a list Let us have a space after ',' and a new line in: - .append(StringUtils.arrayToString(errorFileNames)).append( ". " ); + .append(StringUtils.join( ", " , errorFileNames)).append( ".\n" ); Fix the test code accordingly method verifyTailErrorLogOnContainerExit can/should be private. Same for ContainerExitHandler class. Assume.assumeTrue(Shell.LINUX); should be Assume.assumeFalse(Shell.WINDOWS || Shell.OTHER); but actually why do we need this? The test seems to be platform-independent. Assert.assertNotNull(exitEvent.getDiagnosticInfo()); seems redundant because you then have other asserts implying this already. I suggest to LOG.info the diagnostics instead to make the test log more useful.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the comments Gera Shegalov,

          Things we have not addressed from my previous comments is capping the buffer size.

          Sorry for missing this comment out earlier but as you said its usually small size and default is also small hence we can avoid it.
          Have taken care of other comments in the uploaded patch, please review !

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the comments Gera Shegalov , Things we have not addressed from my previous comments is capping the buffer size. Sorry for missing this comment out earlier but as you said its usually small size and default is also small hence we can avoid it. Have taken care of other comments in the uploaded patch, please review !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 43s trunk passed
          +1 compile 1m 46s trunk passed with JDK v1.8.0_66
          +1 compile 2m 4s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 3m 28s trunk passed
          +1 javadoc 1m 22s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 50s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 21s the patch passed
          +1 compile 1m 44s the patch passed with JDK v1.8.0_66
          +1 javac 1m 44s the patch passed
          +1 compile 2m 5s the patch passed with JDK v1.7.0_91
          +1 javac 2m 5s the patch passed
          -1 checkstyle 0m 26s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 278, now 278).
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 54s the patch passed
          +1 javadoc 1m 21s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 38s the patch passed with JDK v1.7.0_91
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 35s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 7s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          63m 46s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779420/YARN-2934.v2.004.patch
          JIRA Issue YARN-2934
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ebfb287c4704 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6553784
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10090/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10090/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10090/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 43s trunk passed +1 compile 1m 46s trunk passed with JDK v1.8.0_66 +1 compile 2m 4s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 3m 28s trunk passed +1 javadoc 1m 22s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 50s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 21s the patch passed +1 compile 1m 44s the patch passed with JDK v1.8.0_66 +1 javac 1m 44s the patch passed +1 compile 2m 5s the patch passed with JDK v1.7.0_91 +1 javac 2m 5s the patch passed -1 checkstyle 0m 26s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 278, now 278). +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 54s the patch passed +1 javadoc 1m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 38s the patch passed with JDK v1.7.0_91 +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 35s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 9m 7s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 63m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779420/YARN-2934.v2.004.patch JIRA Issue YARN-2934 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ebfb287c4704 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6553784 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10090/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10090/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10090/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          +1 for YARN-2934.v2.004.patch. There is an extra space in "Error files : ", I took the freedom to fix it myself

          Show
          jira.shegalov Gera Shegalov added a comment - +1 for YARN-2934 .v2.004.patch. There is an extra space in "Error files : ", I took the freedom to fix it myself
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9024 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9024/)
          YARN-2934. Improve handling of container's stderr. (Naganarasimha G R (gera: rev 2c17b81569c05b5f375d0ddad99d43cb4e3b23b1)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9024 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9024/ ) YARN-2934 . Improve handling of container's stderr. (Naganarasimha G R (gera: rev 2c17b81569c05b5f375d0ddad99d43cb4e3b23b1) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Committed to trunk and branch-2. Thanks Naganarasimha G R for contribution and patience with reviews. Thanks Vinod Kumar Vavilapalli, Steve Loughran, Varun Saxena, nijel for additional reviews

          Show
          jira.shegalov Gera Shegalov added a comment - Committed to trunk and branch-2. Thanks Naganarasimha G R for contribution and patience with reviews. Thanks Vinod Kumar Vavilapalli , Steve Loughran , Varun Saxena , nijel for additional reviews
          Hide
          Naganarasimha Naganarasimha G R added a comment -
          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review and commit Gera Shegalov , Steve Loughran , Varun Saxena , nijel & Vinod Kumar Vavilapalli .

            People

            • Assignee:
              Naganarasimha Naganarasimha G R
              Reporter:
              jira.shegalov Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development