Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6331 [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)
  3. MAPREDUCE-6720

Inconsistent values of counters across tasks and job reported to timeline service.

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Reviewed


      While testing found below issue. For some of the task counters, we do not have consistent values. This is not the case with every counter though.
      Consider the case of counter "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".

      I found that its value for a flow I ran, was 936018 bytes. For the 3 apps associated with this flow run, the values were 312006 bytes each (which equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further down I found though that for one of the apps, the 4 tasks(2 mappers and 2 reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes each for the 2 mappers.

      This means the value reported for the app should be (2 * 156003 + 2* 155918) or 623842 bytes but it is only 312006 bytes which indicates that only counter value of mappers is being picked up.


        1. Counters for Tasks.png
          198 kB
          Varun Saxena
        2. Counters For Job.png
          326 kB
          Varun Saxena
        3. MAPREDUCE-6720-YARN-2928.01.patch
          1 kB
          Varun Saxena
        4. MAPREDUCE-6720-YARN-2928.02.patch
          6 kB
          Varun Saxena
        5. MAPREDUCE-6720-YARN-2928.03.patch
          6 kB
          Varun Saxena
        6. MAPREDUCE-6720-YARN-2928.04.patch
          4 kB
          Varun Saxena


          This comment will be Viewable by All Users Viewable by All Users


            varun_saxena Varun Saxena Assign to me
            varun_saxena Varun Saxena
            0 Vote for this issue
            8 Start watching this issue




                Issue deployment