Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4562

Support for "FileSystemCounter" legacy counter group name for compatibility reasons is creating incorrect counter name

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.0, 0.23.1, 2.0.0-alpha
    • 0.23.3, 2.0.2-alpha
    • None
    • None
    • Reviewed

    Description

      Hi Guys,
      I was investigating issue in Sqoop project(http://sqoop.apache.org/). Problem is that we are reporting number of written filesystem bytes back to the user and on Hadoop 0.23/2.x we're always getting 0. I've noticed that there was some refactorization in FileSystem counter related code and found MAPREDUCE-3542 requesting backward compatibility.

      Included patch seems to be adding counter "FileSystemCounter":

      hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java:84
      legacyMap.put("FileSystemCounter", FileSystemCounter.class.getName());
      

      But it appears that original name is "FileSystemCounters" (Notice the plural "s" at the end of name):

      src/mapred/org/apache/hadoop/mapred/Task.java:91 (0.20.2)
      protected static final String FILESYSTEM_COUNTER_GROUP = "FileSystemCounters";
      
      src/mapred/org/apache/hadoop/mapred/Task.java:109 (1.0.3)
      protected static final String FILESYSTEM_COUNTER_GROUP = "FileSystemCounters";
      

      I therefore believe that this counter should be renamed in order to provide backward compatibility. I might fix this discrepancy in Sqoop, but I believe that other projects/users might also be affected and therefore it would be better to fix it in upstream.

      Attachments

        1. MAPREDUCE-4562.patch
          3 kB
          Jarek Jarcec Cecho
        2. MAPREDUCE-4562.patch
          2 kB
          Jarek Jarcec Cecho

        Activity

          People

            jarcec Jarek Jarcec Cecho
            jarcec Jarek Jarcec Cecho
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: