Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Include IO offset to client trace logging output.

      Description

      By adding offset in client trace, the client trace information can provide more accurately information about I/O.
      It is useful for performance analyzing.

      Since there is no random write now, the offset of writing is always zero.

      1. clienttrace.patch
        3 kB
        Lei (Eddy) Xu
      2. clienttrace-1.patch
        4 kB
        Lei (Eddy) Xu

        Activity

        Lei (Eddy) Xu created issue -
        Hide
        Lei (Eddy) Xu added a comment -

        adding offset to client trace

        Show
        Lei (Eddy) Xu added a comment - adding offset to client trace
        Lei (Eddy) Xu made changes -
        Field Original Value New Value
        Attachment clienttrace.patch [ 12400028 ]
        Hide
        Chris Douglas added a comment -

        The offset should probably be added after the client id, so the DataNode format remains a superset of the TaskTracker clienttrace. It's probably not useful to add the offset to the shuffle trace, but it's available if you wanted to be consistent.

        Out of curiosity, how are you using the offsets in your performance analysis?

        Show
        Chris Douglas added a comment - The offset should probably be added after the client id, so the DataNode format remains a superset of the TaskTracker clienttrace. It's probably not useful to add the offset to the shuffle trace, but it's available if you wanted to be consistent. Out of curiosity, how are you using the offsets in your performance analysis?
        Hide
        Lei (Eddy) Xu added a comment -

        So far, what I am doing is collecting logs from HDFS for future use. I think the blockId/offset/bytes can provide a accurate records for each I/O operations.

        Thus maybe it will be used for some replay tools and useful for I/O pattern analyzing, however I have not got there yet.

        Show
        Lei (Eddy) Xu added a comment - So far, what I am doing is collecting logs from HDFS for future use. I think the blockId/offset/bytes can provide a accurate records for each I/O operations. Thus maybe it will be used for some replay tools and useful for I/O pattern analyzing, however I have not got there yet.
        Chris Douglas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.21.0 [ 12313563 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12400028/clienttrace.patch
        against trunk revision 744894.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3871/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3871/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3871/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3871/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12400028/clienttrace.patch against trunk revision 744894. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3871/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3871/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3871/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3871/console This message is automatically generated.
        Hide
        Lei (Eddy) Xu added a comment -

        I put offset after client id to keep the DataNode format remaining a superset of the TaskTracker clienttrace.

        And I add DataXceiver into patch.

        Show
        Lei (Eddy) Xu added a comment - I put offset after client id to keep the DataNode format remaining a superset of the TaskTracker clienttrace. And I add DataXceiver into patch.
        Lei (Eddy) Xu made changes -
        Attachment clienttrace-1.patch [ 12400340 ]
        Hide
        Chris Douglas added a comment -

        +1 Looks good

        Show
        Chris Douglas added a comment - +1 Looks good
        Chris Douglas made changes -
        Assignee Lei Xu [ eddyxu ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12400340/clienttrace-1.patch
        against trunk revision 746340.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3891/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3891/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3891/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3891/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12400340/clienttrace-1.patch against trunk revision 746340. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3891/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3891/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3891/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3891/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        I committed this. Thanks, Lei

        Show
        Chris Douglas added a comment - I committed this. Thanks, Lei
        Chris Douglas made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #763 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/763/)
        . Add offset to datanode clienttrace. Contributed by Lei Xu

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #763 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/763/ ) . Add offset to datanode clienttrace. Contributed by Lei Xu
        Owen O'Malley made changes -
        Component/s dfs [ 12310710 ]
        Hide
        Jakob Homan added a comment -

        Adding release note.

        Show
        Jakob Homan added a comment - Adding release note.
        Jakob Homan made changes -
        Release Note Include IO offset to client trace logging output.
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Lei (Eddy) Xu
            Reporter:
            Lei (Eddy) Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development