[HBASE-22120] Replace HTrace with OpenTelemetry

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1
• Component/s: tracing
• Hadoop Flags: Reviewed

Release Note

In this issue we change our tracing system from HTrace to OpenTelemetry.
The HTrace dependencies are banned(transitive dependencies are still allowed as hadoop still depends on them), the imports of htrace related classes are also banned.
We add OpenTelemtry support for our RPC system, which means all the rpc methods will be traced on both client side and server side.
Most methods in Table interface are also traced, except scan and coprocessor related methods. As now the scan implementation is always 'async prefetch', we haven't find a suitable way to represent this relationship between the foreground and background spans yet.
At server side, due to the same reason, we only use a span to record the time of the WAL sync operation, without tracing into the background sync thread.
And we do not trace the next method of RegionScanner, as a scan rpc call may lead to thousands of RegionScanner.next calls, which could slow down the rpc call even when tracing is disabled.
On how to enable tracing, please read the Tracing section in our refguide.
https://hbase.apache.org/book.html#tracing

• Assignee: Duo Zhang
• Reporter: Sergey Shelukhin
• Created: 27/Mar/19 18:15
• Updated: 19/Apr/22 15:33
• Resolved: 25/Apr/21 01:32

Description

Deprecate HTrace usage in HBase

• HBase 1.x (branch-1)
• Declare HTrace (htrace 3.x) deprecated in the user doc.

• HBase 2.x (branch-2)
• Declare HTrace deprecated in the user doc. Furthermore, state that it is known not working.
• Either fix the trace context propagation bug in HBase 2.x, or backport OpenTracing support from the master branch. I am inclined to the latter.

• HBase 3.x (master branch)
• Remove HTrace entirely.
• Add OpenTracing APIs. Potentially backport to HBase 2.4.
• Replace OpenTracing API with OpenTelemetry when the latter stabilizes.

Milestones

1. Doc – deprecation notice
2. Replace existing HTrace code with OpenTracing code in the master branch (3.x) 
3. Java (a poc is currently under way)
4. HBase shell and scripts (Ruby, shell script)
5. Doc 

1. Add new trace instrumentation code for new features not instrumented by the existing HTace code.
2. Propagate the traces to other systems such as HDFS and MapReduce.
3. Support other OpenTracing tracers.

======== Update ========
As OpenTracing has now been replaced by OpenTelemetry, the goal finally becomes replacing HTrace with OpenTelemetry.

Issue Links

• is a parent of: HBASE-26141 Add tracing support for HTable and sync connection on branch-2
• is related to: HBASE-26361 Enable OpenTelemetry to be used from developer sandbox
• is related to: HBASE-26363 OpenTelemetry configuration support for per-process service names
• relates to: HBASE-22115 HBase RPC aspires to grow an infinite tree of trace scopes; some other places are also unsafe
• relates to: HBASE-25810 The nightly build for master branch is failing when running UTs in hbase-procedure after HBASE-22120 merged
• relates to: HBASE-25811 The client integration test is failing after HBASE-22120 merged
• relates to: PHOENIX-5215 Remove and replace HTrace
• relates to: HBASE-24639 RequestId Tracing feature for HBase
• relates to: HADOOP-15566 Support OpenTelemetry
• relates to: HBASE-26419 Tracing Spans of RPCs should follow otel semantic conventions
• relates to: HBASE-26241 Showcase/document a end-to-end tracing integreated with an external sink e.g. Jaeger or zipkin
• relates to: HBASE-25853 Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2
• links to: GitHub Pull Request #2901
• links to: Scope Doc

Sub-Tasks

• [OpenTracing] Declare HTrace is unusable in the user doc: Resolved
• [OpenTracing] Add OpenTracing dependency and helper methods: Resolved
• [OpenTracing] Migrate from HTrace to OpenTracing (Java code): Resolved
• [OpenTracing] Propagate trace context to server correctly: Resolved
• [OpenTracing] Migrate hbase shell and scripts from HTrace to OpenTracing: Resolved
• Add documentation on how to enable and view tracing with OpenTelemetry: Resolved
• [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty: Resolved
• [OpenTracing] Add traces in Procedure V2: Resolved
• Add trace support for simple apis in async client: Resolved
• Remove HTrace completely in code base and try to make use of OpenTelemetry: Resolved
• Add trace support for async call in rpc client: Resolved
• Find a way to config OpenTelemetry tracing without directly depending on opentelemetry-sdk: Resolved
• Add trace support for connection registry: Resolved
• Add trace support for HRegion read/write operation: Resolved
• Add host and port attribute when tracing rpc call at client side: Resolved
• Add trace support for WAL sync: Resolved
• Set span kind to CLIENT in AbstractRpcClient: Resolved
• Upgrade opentelemetry to 0.17.1: Resolved
• Upgrade opentelemetry to 1.0.0: Resolved
• Revisit the span names: Resolved
• Basic benchmark to show the impact on performance for tracing: Resolved
• Temporarily remove the trace support for RegionScanner.next: Resolved
• Change the command line argument for tracing after upgrading opentelemtry to 1.0.0: Resolved
• Upgrade opentelemetry to 1.0.1: Resolved
• The tracinig implementation for AsyncConnectionImpl.getHbck is incorrect: Resolved