Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4562

Support for "FileSystemCounter" legacy counter group name for compatibility reasons is creating incorrect counter name

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 0.23.1, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hi Guys,
      I was investigating issue in Sqoop project(http://sqoop.apache.org/). Problem is that we are reporting number of written filesystem bytes back to the user and on Hadoop 0.23/2.x we're always getting 0. I've noticed that there was some refactorization in FileSystem counter related code and found MAPREDUCE-3542 requesting backward compatibility.

      Included patch seems to be adding counter "FileSystemCounter":

      hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java:84
      legacyMap.put("FileSystemCounter", FileSystemCounter.class.getName());
      

      But it appears that original name is "FileSystemCounters" (Notice the plural "s" at the end of name):

      src/mapred/org/apache/hadoop/mapred/Task.java:91 (0.20.2)
      protected static final String FILESYSTEM_COUNTER_GROUP = "FileSystemCounters";
      
      src/mapred/org/apache/hadoop/mapred/Task.java:109 (1.0.3)
      protected static final String FILESYSTEM_COUNTER_GROUP = "FileSystemCounters";
      

      I therefore believe that this counter should be renamed in order to provide backward compatibility. I might fix this discrepancy in Sqoop, but I believe that other projects/users might also be affected and therefore it would be better to fix it in upstream.

      1. MAPREDUCE-4562.patch
        3 kB
        Jarek Jarcec Cecho
      2. MAPREDUCE-4562.patch
        2 kB
        Jarek Jarcec Cecho

        Activity

          People

          • Assignee:
            Jarek Jarcec Cecho
            Reporter:
            Jarek Jarcec Cecho
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development