[HDFS-1073] Simpler model for Namenode's fs Image and edit Logs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

The NameNode's storage layout for its name directories has been reorganized to be more robust. Each edit now has a unique transaction ID, and each file is associated with a transaction ID (for checkpoints) or a range of transaction IDs (for edit logs).

• Assignee: Todd Lipcon
• Reporter: Sanjay Radia
• Created: 02/Apr/10 05:30
• Updated: 15/Aug/19 05:17
• Resolved: 29/Jul/11 20:33

Description

The naming and handling of NN's fsImage and edit logs can be significantly improved resulting simpler and more robust code.

Issue Links

• is related to: HDFS-955 FSImage.saveFSImage can lose edits
• is related to: HDFS-14378 Simplify the design of multiple NN and both logic of edit log roll and checkpoint

Sub-Tasks

• Refactor edit log loading to a separate class from edit log writing: Closed
• Refactor storage management into separate classes than fsimage file reading/writing: Closed
• Remove intentionally corrupt 0.13 directory layout creation: Closed
• Persist transaction ID on disk between NN restarts: Resolved
• Refactor more startup and image loading code out of FSImage: Resolved
• Add code to detect valid length of an edits file: Resolved
• Add code to inspect a storage directory with txid-based filenames: Resolved
• Add code to list which edit logs are available on a remote NN: Resolved
• Refactor log rolling and filename management out of FSEditLog: Resolved
• reduce need to rewrite fsimage on statrtup: Resolved
• Extend image checksumming to function with multiple fsimage files: Resolved
• Remove use of timestamps to identify checkpoints and logs: Resolved
• Add migration tests from old-format to new-format storage: Resolved
• Add state management variables to FSEditLog: Resolved
• Add some convenience functions to iterate over edit log streams: Resolved
• Update HDFS-1073 branch to deal with OP_INVALID-filled preallocation: Resolved
• Change edit logs and images to be named based on txid: Resolved
• Add constants for LAYOUT_VERSIONs in edits log branch: Resolved
• Additional QA tasks for Edit Log branch: Resolved
• Remove references to StorageDirectory from JournalManager interface: Resolved
• TestDFSUpgrade failing in HDFS-1073 branch: Resolved
• HDFS-1073: Fix backupnode for new edits/image layout: Resolved
• 1073: Enable multiple checkpointers to run simultaneously: Resolved
• HDFS-1073: Cleanup in image transfer servlet: Resolved
• HDFS-1073: Test for 2NN downloading image is not running: Resolved
• HDFS-1073: Some refactoring of 2NN to easier share code with BN and CN: Resolved
• Remove vestiges of NNStorageListener: Resolved
• TestCheckpoint needs to clean up between cases: Resolved
• Fix race conditions when running two rapidly checkpointing 2NNs: Resolved
• Image transfer process misreports client side exceptions: Resolved
• HDFS-1073: Kill previous.checkpoint, lastcheckpoint.tmp directories: Resolved
• Clean up and test behavior under failed edit streams: Resolved
• 1073: Remove checkpointTxId from VERSION file: Resolved
• 1073: remove/archive unneeded/old storage files: Resolved
• 1073: 2NN needs to handle case of reformatted NN better: Resolved
• 1073: Image inspector should return finalized logs before unfinalized logs: Resolved
• 1073: Improve TestNamespace and TestEditLog in 1073 branch: Resolved
• 1073: Improve upgrade tests from 0.22: Resolved
• 1073: determine edit log validity by truly reading and validating transactions: Resolved
• 1073: address checkpoint upload when one of the storage dirs is failed: Resolved
• 1073: NN should not clear storage directory when restoring removed storage: Resolved
• 1073: create an escape hatch to ignore startup consistency problems: Resolved
• 1073: finalize inprogress edit logs at startup: Resolved
• 1073: Move edits log archiving logic into FSEditLog/JournalManager: Resolved
• 1073: Handle case where an entirely empty log is left during NN crash: Resolved
• 1073: consider adding END_LOG_SEGMENT txn when finalizing inprogress logs at startup: Open
• 1073: update remaining unit tests to new storage filenames: Resolved
• 1073: Add a flag to 2NN to format its checkpoint dirs on startup: Resolved
• 1073: Checkpoint interval should be based on txn count, not size: Resolved
• 1073: address remaining TODOs and pre-merge cleanup: Resolved
• 1073: fix regression of HDFS-1955 in branch: Resolved
• 1073: Fault injection for StorageDirectory failures during read/write of FSImage/Edits files: Open
• 1073: Zero pad edits filename to make them lexically sortable: Resolved
• 1073: Move all journal stream management code into one place: Closed
• 1073: fix CreateEditsLog test tool in branch: Resolved
• 1073: Reenable TestEditLog.testFailedOpen and fix exposed bug: Resolved
• 1073: clean up TestCheckpoint and remove TODOs: Resolved
• 1073: Address remaining TODOs: Resolved
• 1073: address findbugs/javadoc warnings: Resolved
• saveNamespace should not throw IOE when only one storage directory fails to write VERSION file: Resolved
• Complete decoupling of failure states between edits and image dirs: Open