[HDFS-14276] [SBN read] Reduce tailing overhead

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 2.10.0, 3.3.0, 3.2.1, 3.1.3
• Component/s: ha, namenode

Environment

Hardware: 4-node cluster, each node has 4 core, Xeon 2.5Ghz, 25GB memory.
Software: CentOS 7.4, CDH 6.0 + Consistent Reads from Standby, Kerberos, SSL, RPC encryption + Data Transfer Encryption.

• Hadoop Flags: Reviewed
• Assignee: Ayush Saxena
• Reporter: Wei-Chiu Chuang
• Created: 13/Feb/19 23:49
• Updated: 02/Oct/19 17:15
• Resolved: 21/Aug/19 21:04

Description

When Observer sets dfs.ha.tail-edits.period = 0ms, it tails edit log continuously in order to fetch the latest edits, but there is a lot of overhead in doing so.

Critically, edit log tailer should not update NameDirSize metric every time. It has nothing to do with fetching edits, and it involves lots of directory space calculation.

Profiler suggests a non-trivial chunk of time is spent for nothing.

Other than this, the biggest overhead is in the communication to serialize/deserialize messages to/from JNs. I am looking for ways to reduce the cost because it's burning 30% of my CPU time even when the cluster is idle.

Issue Links

• relates to: HDFS-14370 Edit log tailing fast-path should allow for backoff
• relates to: HDFS-14277 [SBN read] Observer benchmark results