Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1851

FileSystem counters do not differentiate between different FileSystems

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.5.3
    • Fix Version/s: 0.5.4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      I checked hadoop counters after running a hive query on tez. The logs show two counters for BYTES_READ but does not differentiate between them. Here is the log line for counters

      2014-12-12 15:17:10,102 INFO [AsyncDispatcher event handler] history.HistoryEventHandler: [HISTORY][DAG:dag_1418425019273_0003_1][Event:DAG_FINISHED]: dagId=dag_1418425019273_0003_1, startTime=1418426223988, finishTime=1418426230084, timeTaken=6096, status=SUCCEEDED, diagnostics=, counters=Counters: 49, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=7, TOTAL_LAUNCHED_TASKS=7, DATA_LOCAL_TASKS=6, File System Counters, BYTES_READ=250, BYTES_WRITTEN=358, READ_OPS=0, LARGE_READ_OPS=0, WRITE_OPS=0, BYTES_READ=7918626, BYTES_WRITTEN=8, READ_OPS=38, LARGE_READ_OPS=0, WRITE_OPS=2, 
      
      1. TEZ-1851.1.txt
        0.6 kB
        Siddharth Seth

        Activity

        Hide
        sseth Siddharth Seth added a comment -

        We seem to have lost the differentiation between Local / HDFS. Setting target to 0.6.0.

        Show
        sseth Siddharth Seth added a comment - We seem to have lost the differentiation between Local / HDFS. Setting target to 0.6.0.
        Hide
        sseth Siddharth Seth added a comment -

        Trivial patch. Prakash Ramachandran - please take a look.

        Show
        sseth Siddharth Seth added a comment - Trivial patch. Prakash Ramachandran - please take a look.
        Hide
        sseth Siddharth Seth added a comment -

        We should probably pull this into 0.5.4 as well.

        Show
        sseth Siddharth Seth added a comment - We should probably pull this into 0.5.4 as well.
        Hide
        hitesh Hitesh Shah added a comment -

        Does this need to be changed elsewhere too ( whereever the display name is different from the actual name ) ?

        Show
        hitesh Hitesh Shah added a comment - Does this need to be changed elsewhere too ( whereever the display name is different from the actual name ) ?
        Hide
        sseth Siddharth Seth added a comment -

        This is what it should be for the other implementations of AbstractCounters.

        Show
        sseth Siddharth Seth added a comment - This is what it should be for the other implementations of AbstractCounters.
        Hide
        hitesh Hitesh Shah added a comment -

        Sorry - should have clarified. Do we need to fix any other counters in the tez code?

        Show
        hitesh Hitesh Shah added a comment - Sorry - should have clarified. Do we need to fix any other counters in the tez code?
        Hide
        sseth Siddharth Seth added a comment -

        Don't think I understand the question. The display name and actual name should be the same for all counters now... even though we do have a constructor which accepts different values. If that is an issue, it would be a separate jira.

        Show
        sseth Siddharth Seth added a comment - Don't think I understand the question. The display name and actual name should be the same for all counters now... even though we do have a constructor which accepts different values. If that is an issue, it would be a separate jira.
        Hide
        pramachandran Prakash Ramachandran added a comment - - edited

        +1 lgtm. checked the counters names and counter display names are same for all counters.

        one quick question though - for the counter group name they seem to be different for ex
        name: org.apache.tez.common.counters.FileSystemCounter and counterGroupDisplayName: "File System Counters". should this be the same?

        Show
        pramachandran Prakash Ramachandran added a comment - - edited +1 lgtm. checked the counters names and counter display names are same for all counters. one quick question though - for the counter group name they seem to be different for ex name: org.apache.tez.common.counters.FileSystemCounter and counterGroupDisplayName: "File System Counters". should this be the same?
        Hide
        sseth Siddharth Seth added a comment -

        On the FileSystemCounter group name - I think it's ok to leave the display name in place, since this is a specialized counter. I'm a little worried about compatibility. What happens when counters are fetched by the TezClient from ATS - will the display name and name be available ?

        Show
        sseth Siddharth Seth added a comment - On the FileSystemCounter group name - I think it's ok to leave the display name in place, since this is a specialized counter. I'm a little worried about compatibility. What happens when counters are fetched by the TezClient from ATS - will the display name and name be available ?
        Hide
        pramachandran Prakash Ramachandran added a comment -

        yes the displayname and name are available in DAGClientTimelineImpl data fetched from ATS. not sure if i got the concern right, but they are used only for display purpose (displayname is used in the status print).

        Show
        pramachandran Prakash Ramachandran added a comment - yes the displayname and name are available in DAGClientTimelineImpl data fetched from ATS. not sure if i got the concern right, but they are used only for display purpose (displayname is used in the status print).
        Hide
        sseth Siddharth Seth added a comment -

        Alright. I'm going to go ahead and commit this then. Thanks for the review. Also pulling into 0.5.4.

        Show
        sseth Siddharth Seth added a comment - Alright. I'm going to go ahead and commit this then. Thanks for the review. Also pulling into 0.5.4.
        Hide
        sseth Siddharth Seth added a comment -

        Committed to master and branch-0.5

        Show
        sseth Siddharth Seth added a comment - Committed to master and branch-0.5
        Hide
        hitesh Hitesh Shah added a comment -

        Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

        Show
        hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

          People

          • Assignee:
            sseth Siddharth Seth
            Reporter:
            prasanth_j Prasanth Jayachandran
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development