[HDFS-5274] Add Tracing to HDFS

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1-beta
• Component/s: datanode, namenode
• Labels: BB2015-05-TBR
• Assignee: Elliott Neil Clark
• Reporter: Elliott Neil Clark
• Created: 28/Sep/13 00:52
• Updated: 25/Jun/19 19:46
• Resolved: 11/Aug/15 17:05

Description

Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS.

Issue Links

• is depended upon by: ACCUMULO-1197 Pass Accumulo trace functionality through the DFSClient
• is related to: HBASE-6449 Dapper like tracing
• is related to: HADOOP-15566 Support OpenTelemetry
• relates to: HDFS-7001 Tests in TestTracing should not depend on the order of execution

Sub-Tasks

• Adding tracing to Hadoop RPC: Closed
• Adding tracing to DataNode data transfer protocol: Closed
• Allow dynamically changing the tracing level in Hadoop servers: Closed
• Allow DFSClient to manually specify that a request should be traced: Resolved
• Add tracing to DFSInputStream: Closed
• The DFSClient should use the sampler to determine whether to initiate trace spans when making RPCv9 calls to the NN and DN: Resolved
• Add trace spans for DFSClient metadata operations: Closed
• Make DFSOutputStream tracing more fine-grained: Closed
• Add trace command to FsShell: Resolved
• Add htrace configuration properties to core-default.xml and update user doc about how to enable htrace: Closed
• Add tracing to DFSClient#setQuotaByStorageType: Closed