Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3843

Job summary log file found missing on the RM host

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: jobhistoryserver, mrv2
    • Labels:
      None

      Description

      This bug was found by Phil Su as part of our testing.

      After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
      RM host.

      The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
      e.g.
      12/02/09 15:57:21 INFO jobhistory.JobSummary:
      jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
      esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0

      1) On the RM with older hadoop version where the job summary log does not exist
      mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
      -Djava.net.preferIPv4Stack=true
      -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
      home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
      -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
      -Dhadoop
      .root.logger=INFO,console
      -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
      ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
      -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer

      2) On the RM with older hadoop version where the job summary log exists
      mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
      -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
      -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
      -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
      -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
      Dyarn.log.file=yarn-mapredqa-historyserver<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
      -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
      -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
      -Dyarn.log.dir=/home/gs/var/log/mapredqa
      Dhadoop.log.file=yarn-mapredqa-historyserver<host>.log
      Dyarn.log.file=yarn-mapredqa-historyserver<host>.log
      -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
      -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
      /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/:/home/gs/gridre/shelob/share/hadoop/hadoop-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoopcapacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
      org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer

      1) On the RM with older hadoop version where the job summary log does not exist
      jobhistory ps shows using the option:
      -Dmapred.jobsummary.logger=INFO,console

      2) On the RM with older hadoop version where the job summary log exists
      jobhistory ps shows using the option:
      -Dmapred.jobsummary.logger=INFO,JSA
      -Dmapred.jobsummary.logger=INFO,JSA

      1. MAPREDUCE-3843.patch
        8 kB
        Thomas Graves
      2. MAPREDUCE-3843.patch
        7 kB
        Anupam Seth
      3. MAPREDUCE-3843.patch
        7 kB
        Anupam Seth
      4. MAPREDUCE-3843.patch
        3 kB
        Anupam Seth
      5. MAPREDUCE-3843.patch
        2 kB
        Anupam Seth
      6. MAPREDUCE-3843.patch
        0.7 kB
        Anupam Seth

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Anupam Seth
              Reporter:
              Anupam Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development