Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5453

File System Counters can NOT be updated in streaming application

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It seems that FILE_BYTES_READ/FILE_BYTES_WRITTEN can NOT be updated in streaming application by:

      reporter:counter:<group>,<counter>,<amount>

      The syntax I tried is:
      System.err.println("reporter:counter:"+ "FileSystemCounters" + ","
      + "FILE_BYTES_READ" + ","
      + number);

      Since File System counters are added later into the framework by ticket
      https://issues.apache.org/jira/browse/HADOOP-3001

      and that's why it is NOT properly handled by streaming.jar.

      The implementation of File System counter is a bit different, and it keeps track of the real information in FileSystem.Statistics object. When the task finished, the framework would read the info from 'FileSystem.Statistics' and set the File System counters accordingly.

      Therefore, the regular way to update counters – ' findCounter() and Counter.incrment() ' would NOT work for the File System counters as they would be overwritten by the info in 'FileSystem.Statistics ' eventually.
      However, it seems that the 'reporter:counter:' scheme in streaming.jar is implemented by the regular way, and maybe that's why it is not working.

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Danica Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development