Dapper is google's system to trace requests through their systems.
The paper discusses how useful Google engineers find the system, as well as laying out the basic design of the system.

Accumulo added tracing to their project based on the Dapper design. They call it cloudtrace. I have used Accumulo's cloudtrace as a starting point for a tracing library. I modified it to not be dependent on any particular RPC framework, as well as adding more detailed sampling, among other changes. I plan to commit this to the hbase source tree as an independent maven project. I will then submit commits that properly instrument HBase to use the tracing library. This requires adding an optional payload to the protobuf RPC's containing two 64bit id's: one for the trace id and another for the unique id of the span that sent the message. There will be no additional data in the messages when tracing is not active. The tracing library works by storing information in ThreadLocal's. In order to preserve this information across thread changes, the thread changes must be tracked. The tracing package accomplishes this by wrapping any thread changes to pass the information through to the new thread. I have done most of the instrumentation for HBase.

This is just a sample trace that I made with my instrumented version of hbase. It shows the operations involved in a "delete table" operation. To create this, I had the tracing library write the spans to local files, which I piped into a simple python script to make a graphviz graph.

Woah is it me or we really are doing a lot of .META. scans in there?

Wow

Very useful stuff here.

@Jonathan:
HBase trunk now has several maven modules.
Is it possible to format your contribution as a new maven module ?

Thanks

@Zhihong I was planning to keep the library as independent as possible so it can be broken out easily later and used for other hadoop systems. What are your thoughts on this?

Sounds good to me.

This is the initial commit of the htrace library.

Link to review board for htrace1.diff.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537786/htrace1.diff
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 8 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2434//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2434//console

This message is automatically generated.

Excellent

You're right about test categorization.

Please add htrace module to the root pom.xml (this was the reason why htrace tests were not run):

Index: pom.xml
===================================================================
--- pom.xml	(revision 1365854)
+++ pom.xml	(working copy)
@@ -46,6 +46,7 @@
     <module>hbase-hadoop-compat</module>
     <module>hbase-common</module>
     <module>hbase-it</module>
+    <module>htrace</module>
   </modules>
   <scm>
     <connection>scm:svn:http://svn.apache.org/repos/asf/hbase/trunk</connection>

Then you should be able to run test individually at workspace root:

Running org.apache.hadoop.hbase.htrace.CountSamplerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.039 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] HBase ............................................. SUCCESS [1.479s]
[INFO] HBase - Common .................................... SUCCESS [0.792s]
[INFO] HBase - Hadoop Compatibility ...................... SUCCESS [0.089s]
[INFO] HBase - Hadoop One Compatibility .................. SUCCESS [0.235s]
[INFO] HBase - Server .................................... SUCCESS [3.212s]
[INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [1.581s]
[INFO] HBase - Integration Tests ......................... SUCCESS [0.376s]
[INFO] htrace ............................................ SUCCESS [0.553s]

Please change the name of htrace to 'HBase - Trace' so that it aligns with other module names.

Looks like you forgot to include a file in the patch:

Running org.apache.hadoop.hbase.htrace.TestHTrace
Error constructing LocalFileSpanReceiver: test/htrace-test-output-spans.txt (No such file or directory)

@zhihong
Do you think it is best to add it as a module as opposed to keeping it completely independent, and just having hbase depend on it?

A completely independent project is good to have but requires more logistics.
We can refine the htrace module using this JIRA so that HBase is able to use it.

Then we can propose the promotion of this feature as an independent project.

Sounds good. I will submit a new patch today that deals with most of your and Todd's changes as well as making htrace a module of hbase. Let's just be careful going forward to ensure that the tracing library can easily be broken out when necessary.

Do you think it is best to add it as a module as opposed to keeping it completely independent, and just having hbase depend on it?

I think it best to have it as an independent project, at least at first (you don't need our blessing to do such a thing). Make a patch for hbase core that gives you the hooks you need and we'll commit that but better if all the rest can live outside especially as its being developed (no lag waiting on a committer to review and commit; just do it yourself out in github or wherever).

What do you mean by having hbase dependent on htrace?

What do you mean by having hbase dependent on htrace?

I meant having htrace as an independent maven project in the hbase source tree and add a <dependency> for htrace in hbase's pom.
The original plan was to get the tracing library committed to hbase first so the hbase committers could review and would be more likely to commit the necessary hooks into hbase. It would also save the initial hassle of getting the library into maven.

But you would recommend putting it on github, and only submitting patches with the instrumentation for hbase? If htrace were on github and the hooks were in hbase, would those downloading hbase source just have to grab htrace off github and have maven install it locally?

I updated the diff on review board with many of Todd and Ted's suggestions.

I meant having htrace as an independent maven project in the hbase source tree and add a <dependency> for htrace in hbase's pom.

But sounds like your intent is a generic tracing library going by statements above? Is that so? If that is the case, it seems odd dev'ing a generic lib inside in HBase core. You should do it as its own project I'd say; you can move faster and it'll be easier keep your generic focus. HBase project, IMO, will gladly be your guinea pig/proofing ground for your tracing lib work since we need tracing so bad. We can review/exercise your lib, we can add it as a dependency and make the necessary changes to pull it in and make it easy starting/stopping a trace, etc. but unless you think otherwise, lets keep the coupling loose.

Correct me if I'm off Jonathan.

One more thing, I'd say HBase wants the generic lib. An hbase-specific would be great but won't be as effective as one that does NOT stop dead in its tracks at the HDFS API. A trace that goes all the ways down would be the cats pyjamas.

@stack you nailed it. The original goal was just tracing for hbase, but it seemed silly to not put in the extra effort necessary to keep the tracing code generic enough for use elsewhere.

So it seems the consensus is that I should get the tracing code up on github and develop it there? Then I would submit patches that just integrate the library into htrace?

@Jonathan Sounds good to me. Maybe others have opinions otherwise. Would suggest that you keep running commentary on your progress, say here in this issue, because there is a bunch of interest in making sure this works for hbase so I for one would be up for review and trying it out whats up on github or wherever (and as said already, for getting in hooks you need – perhaps in a different issue – and its ok if you don't nail it the first time... we can file new issues to adjust as you learn more what you need).

Second version of the patch. Also available on the review board. Reworking the API today.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538371/htrace2.diff
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 17 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2456//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2456//console

This message is automatically generated.

Just a progress update:
The tracing library is up on github . I am close to having it in maven central, which will make it easy to add to hbase. There are some big API changes coming soon (today?) and once those are done I will try to do the first maven release.

I am preparing an hbase patch with the hooks I have now that I will hopefully upload for review early next week.

@Jonathan Sounds good. Would suggest new issue for adding trace hooks to hbase. Could be subissue of this one. Good on you.

Just made a subissue of this one that has a diff with the hooks needed for tracing rpcs. The diff is also on review board.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538371/htrace2.diff
against trunk revision .
ATTACHMENT ID: 12538371

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 17 new or modified tests.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9055//console

This message is automatically generated.