[HDFS-6185] HDFS operational and debuggability improvements

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Suresh Srinivas
• Reporter: Suresh Srinivas
• Created: 02/Apr/14 17:18
• Updated: 04/Sep/14 23:11

Description

This umbrella jira proposes improvements in HDFS to make operations simpler and the system more robust. The areas of improvements includes - ensuring configuration is correct and follows the know best practices, make HDFS robust against some of the failures observed due to cluster not being monitored correctly, better metadata management to avoid lengthy startup time etc.

Subtasks will be filed with more details on individual improvements.

The goal is simplify operations, improve debuggability and robustness. If you have ideas on improvements that fall under this umbrella, please file subtasks under this jira.

Issue Links

• is related to: HDFS-6982 nntop: top­-like tool for name node users

Sub-Tasks

• Provide diagnosis information in the Web UI: Patch Available
• Capture the number of blocks pending deletion on namenode webUI: Closed
• Pause deletion of blocks when the namenode starts up: Closed
• Save namespace when the namenode is stopped: Open
• Check and make checkpoint before stopping the NameNode: Resolved
• Show when block deletion will start after NameNode startup in WebUI: Closed
• Improve diagnostics on storage directory rename operations by using NativeIO#renameTo in Storage#rename.: Closed
• Split error checks in AtomicFileOutputStream#close into separate conditions to improve diagnostics.: Closed
• When aborting NameNode or JournalNode due to metadata file problems, write the contents of the metadata directories and permissions to logs.: Open
• For JournalNode operations that must succeed on all nodes, execute a pre-check to verify that the operation can succeed.: Open
• Adding additional tracing for memory storage eviction algorithm: Open
• Improve the log for HA failover: Open
• Add a counter in the log that shows the number of block reports processed: Patch Available
• Add additional trace to DataNode#shutdown to track progress: Open
• Improve retry attempt log in DFSOutputStream#completeFile with current attempt number: Patch Available