[HDFS-7129] Metrics to track usage of memory for writes

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HDFS-6581
• Fix Version/s: 2.6.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Xiaoyu Yao
• Reporter: Arpit Agarwal
• Created: 22/Sep/14 23:06
• Updated: 22/May/17 12:52
• Resolved: 30/Sep/14 07:54

Description

A few metrics to evaluate feature usage and suggest improvements. Thanks to sureshms for some of these suggestions.

1. Number of times a block in memory was read (before being ejected)
2. Average block size for data written to memory tier
3. Time the block was in memory before being ejected
4. Number of blocks written to memory
5. Number of memory writes requested but not satisfied (failed-over to disk)
6. Number of blocks evicted without ever being read from memory
7. Average delay between memory write and disk write (window where a node restart could cause data loss).
8. Replicas written to disk by lazy writer
9. Bytes written to disk by lazy writer
10. Replicas deleted by application before being persisted to disk

Issue Links

• relates to: HDFS-11864 Document Metrics to track usage of memory for writes