Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-75

Job history log does not output final counters if job is failed or killed

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The last log entry in job history contains important counters. This information is missing if the job is killed or failed.

        Activity

        Hide
        Harsh J added a comment -

        In MR2's Job History Server, I can note that the history does carry aggregate counters from the non-killed tasks of a job, regardless of the job itself passing/failing/getting killed.

        Hence, I am resolving this as Cannot Reproduce (anymore).

        Show
        Harsh J added a comment - In MR2's Job History Server, I can note that the history does carry aggregate counters from the non-killed tasks of a job, regardless of the job itself passing/failing/getting killed. Hence, I am resolving this as Cannot Reproduce (anymore).
        Hide
        Mac Yang added a comment -

        Sample output from a normal job,

        Job JOBID="job_200904200538_0393" FINISH_TIME="1240248187540" JOB_STATUS="SUCCESS" FINISHED_MAPS="20" FINISHED_REDUCES="1" FAILED_MAPS="0" FAILED_REDUCES="0" COUNTERS="

        {(org\.apache\.hadoop\.mapred\.JobInProgress$Counter)(Job Counters )[(TOTAL_LAUNCHED_REDUCES)(Launched reduce tasks)(1)][(RACK_LOCAL_MAPS)(Rack-local map tasks)(2)][(TOTAL_LAUNCHED_MAPS)(Launched map tasks)(20)][(DATA_LOCAL_MAPS)(Data-local map tasks)(1)]} {(FileSystemCounters)(FileSystemCounters)[(FILE_BYTES_READ)(FILE_BYTES_READ)(83248)][(HDFS_BYTES_READ)(HDFS_BYTES_READ)(2019250)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(948688)][(HDFS_BYTES_WRITTEN)(HDFS_BYTES_WRITTEN)(56630)]} {(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(REDUCE_INPUT_GROUPS)(Reduce input groups)(5315)][(COMBINE_OUTPUT_RECORDS)(Combine output records)(0)][(MAP_INPUT_RECORDS)(Map input records)(72020)][(REDUCE_SHUFFLE_BYTES)(Reduce shuffle bytes)(821560)][(REDUCE_OUTPUT_RECORDS)(Reduce output records)(5315)][(SPILLED_RECORDS)(Spilled Records)(1057000)][(MAP_OUTPUT_BYTES)(Map output bytes)(3373000)][(MAP_INPUT_BYTES)(Map input bytes)(2965720)][(MAP_OUTPUT_RECORDS)(Map output records)(528500)][(COMBINE_INPUT_RECORDS)(Combine input records)(0)][(REDUCE_INPUT_RECORDS)(Reduce input records)(528500)]}

        " .

        Sample output from a killed job,

        Job JOBID="job_200904200538_0015" FINISH_TIME="1240209608007" JOB_STATUS="KILLED" FINISHED_MAPS="1" FINISHED_REDUCES="0" .

        Show
        Mac Yang added a comment - Sample output from a normal job, Job JOBID="job_200904200538_0393" FINISH_TIME="1240248187540" JOB_STATUS="SUCCESS" FINISHED_MAPS="20" FINISHED_REDUCES="1" FAILED_MAPS="0" FAILED_REDUCES="0" COUNTERS=" {(org\.apache\.hadoop\.mapred\.JobInProgress$Counter)(Job Counters )[(TOTAL_LAUNCHED_REDUCES)(Launched reduce tasks)(1)][(RACK_LOCAL_MAPS)(Rack-local map tasks)(2)][(TOTAL_LAUNCHED_MAPS)(Launched map tasks)(20)][(DATA_LOCAL_MAPS)(Data-local map tasks)(1)]} {(FileSystemCounters)(FileSystemCounters)[(FILE_BYTES_READ)(FILE_BYTES_READ)(83248)][(HDFS_BYTES_READ)(HDFS_BYTES_READ)(2019250)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(948688)][(HDFS_BYTES_WRITTEN)(HDFS_BYTES_WRITTEN)(56630)]} {(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(REDUCE_INPUT_GROUPS)(Reduce input groups)(5315)][(COMBINE_OUTPUT_RECORDS)(Combine output records)(0)][(MAP_INPUT_RECORDS)(Map input records)(72020)][(REDUCE_SHUFFLE_BYTES)(Reduce shuffle bytes)(821560)][(REDUCE_OUTPUT_RECORDS)(Reduce output records)(5315)][(SPILLED_RECORDS)(Spilled Records)(1057000)][(MAP_OUTPUT_BYTES)(Map output bytes)(3373000)][(MAP_INPUT_BYTES)(Map input bytes)(2965720)][(MAP_OUTPUT_RECORDS)(Map output records)(528500)][(COMBINE_INPUT_RECORDS)(Combine input records)(0)][(REDUCE_INPUT_RECORDS)(Reduce input records)(528500)]} " . Sample output from a killed job, Job JOBID="job_200904200538_0015" FINISH_TIME="1240209608007" JOB_STATUS="KILLED" FINISHED_MAPS="1" FINISHED_REDUCES="0" .

          People

          • Assignee:
            Unassigned
            Reporter:
            Mac Yang
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development