Chukwa
  1. Chukwa
  2. CHUKWA-282

Demux should output detailed per-operation ClientTrace records

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.2.0
    • Fix Version/s: 0.2.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      Currently, the ClientTrace reducer in Demux summarizes client trace operations to produce aggregate statistics for each type of operation (e.g. total bytes of local HDFS reads, total bytes of local HDFS writes). However, more sophisticated algorithms and visualization will need more detailed information such as amount of reads/writes to/from every pair of hosts. The ClientTrace reducer should also output the unsummarized, raw ClientTrace data so they can be used for analysis/visualization.

        Activity

        Jiaqi Tan created issue -
        Hide
        Jiaqi Tan added a comment -

        ClientTraceProcessor adds an additional "actual_time" field to the ChukwaRecord for generating ChukwaRecordKey of ClientTraceDetailed ChukwaRecords, and ClientTrace reducer generates additional ClientTraceDetailed records.

        Show
        Jiaqi Tan added a comment - ClientTraceProcessor adds an additional "actual_time" field to the ChukwaRecord for generating ChukwaRecordKey of ClientTraceDetailed ChukwaRecords, and ClientTrace reducer generates additional ClientTraceDetailed records.
        Jiaqi Tan made changes -
        Field Original Value New Value
        Attachment CHUKWA-282.patch [ 12410244 ]
        Jiaqi Tan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ari Rabkin added a comment -

        Looks okay to me, but I don't feel like I know that part of the code well enough to be 100% sure, so I'd prefer if Eric or Jerome gave it a +1.

        Also. Is it practical to unit-test this? We like to get new unit tests for each nontrivial change to the Java code.

        Show
        Ari Rabkin added a comment - Looks okay to me, but I don't feel like I know that part of the code well enough to be 100% sure, so I'd prefer if Eric or Jerome gave it a +1. Also. Is it practical to unit-test this? We like to get new unit tests for each nontrivial change to the Java code.
        Hide
        Jiaqi Tan added a comment -

        I think a simple unit test would be reasonable in this case, just to sanity check the code. It's essentially just moving all the key/value pairs in each cilenttrace message into the final post-Demux output. The unit test should be pretty trivial but would make sense in any case.

        Show
        Jiaqi Tan added a comment - I think a simple unit test would be reasonable in this case, just to sanity check the code. It's essentially just moving all the key/value pairs in each cilenttrace message into the final post-Demux output. The unit test should be pretty trivial but would make sense in any case.
        Hide
        Eric Yang added a comment -

        Jiaqi, could you add the unit test?

        Show
        Eric Yang added a comment - Jiaqi, could you add the unit test?
        Hide
        Jiaqi Tan added a comment -

        Yes, I expect to be able to get it done by early next week. Sorry for the delay.

        Show
        Jiaqi Tan added a comment - Yes, I expect to be able to get it done by early next week. Sorry for the delay.
        Hide
        Ari Rabkin added a comment -

        Jiaqi – you mentioned a revised version with unit tests. Is that likely to happen? Should I cancel the current patch, or commit it?

        Show
        Ari Rabkin added a comment - Jiaqi – you mentioned a revised version with unit tests. Is that likely to happen? Should I cancel the current patch, or commit it?
        Hide
        Ari Rabkin added a comment -

        I just committed this; thanks Jiaqi.

        Show
        Ari Rabkin added a comment - I just committed this; thanks Jiaqi.
        Ari Rabkin made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Jiaqi Tan
            Reporter:
            Jiaqi Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development