[HDFS-10285] Storage Policy Satisfier in HDFS

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HDFS-10285
• Fix Version/s: HDFS-10285, 3.2.0
• Component/s: datanode, namenode

Release Note

StoragePolicySatisfier(SPS) allows users to track and satisfy the storage policy requirement of a given file/directory in HDFS. User can specify a file/directory path by invoking “hdfs storagepolicies -satisfyStoragePolicy -path <path>” command or via HdfsAdmin#satisfyStoragePolicy(path) API. For the blocks which has storage policy mismatches, it moves the replicas to a different storage type in order to fulfill the storage policy requirement. Since API calls goes to NN for tracking the invoked satisfier path(iNodes), administrator need to enable dfs.storage.policy.satisfier.mode’ config at NN to allow these operations. It can be enabled by setting ‘dfs.storage.policy.satisfier.mode’ to ‘external’ in hdfs-site.xml. The configs can be disabled dynamically without restarting Namenode. SPS should be started outside Namenode using "hdfs --daemon start sps". If administrator is looking to run Mover tool explicitly, then he/she should make sure to disable SPS first and then run Mover. See the "Storage Policy Satisfier (SPS)" section in the Archival Storage guide for detailed usage.

• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 14/Apr/16 02:21
• Updated: 21/Sep/18 11:43
• Resolved: 21/Sep/18 11:38

Description

Heterogeneous storage in HDFS introduced the concept of storage policy. These policies can be set on directory/file to specify the user preference, where to store the physical block. When user set the storage policy before writing data, then the blocks could take advantage of storage policy preferences and stores physical block accordingly.

If user set the storage policy after writing and completing the file, then the blocks would have been written with default storage policy (nothing but DISK). User has to run the ‘Mover tool’ explicitly by specifying all such file names as a list. In some distributed system scenarios (ex: HBase) it would be difficult to collect all the files and run the tool as different nodes can write files separately and file can have different paths.

Another scenarios is, when user rename the files from one effected storage policy file (inherited policy from parent directory) to another storage policy effected directory, it will not copy inherited storage policy from source. So it will take effect from destination file/dir parent storage policy. This rename operation is just a metadata change in Namenode. The physical blocks still remain with source storage policy.

So, Tracking all such business logic based file names could be difficult for admins from distributed nodes(ex: region servers) and running the Mover tool.

Here the proposal is to provide an API from Namenode itself for trigger the storage policy satisfaction. A Daemon thread inside Namenode should track such calls and process to DN as movement commands.

Will post the detailed design thoughts document soon.

Issue Links

• is depended upon by: HDFS-12226 Follow-on work for Storage Policy Satisfier in Namenode
• relates to: HDFS-11257 Evacuate DN when the remaining is negative

Sub-Tasks

• [SPS]: Provide storage policy satisfy worker at DN for co-ordinating the block storage movement work: Resolved
• [SPS]: Daemon thread in Namenode to find blocks placed in other storage than what the policy specifies: Resolved
• [SPS]: Protocol buffer changes for sending storage movement commands from NN to DN: Resolved
• [SPS]: Add satisfyStoragePolicy API in HdfsAdmin: Resolved
• [SPS]: Add block movement tracker to track the completion of block movement future tasks at DN: Resolved
• [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on: Resolved
• [SPS]: Provide mechanism to send blocks movement result back to NN from coordinator DN: Resolved
• [SPS]:Provide retry mechanism for the blocks which were failed while moving its storage at DNs: Resolved
• [SPS]: Handling of block movement failure at the coordinator datanode: Resolved
• [SPS]: Provide unique trackID to track the block movement sends to coordinator: Resolved
• [SPS] Make storage policy satisfier daemon work on/off dynamically: Resolved
• [SPS]: StoragePolicySatisfier should gracefully handle when there is no target node with the required storage type: Resolved
• [SPS]: Handle partial block location movements: Resolved
• [SPS]: Erasure coded files should be considered for satisfying storage policy: Resolved
• [SPS]: Make SPS movement monitor timeouts configurable: Resolved
• [SPS]: Local DN should be given preference as source node, when target available in same node: Resolved
• [SPS]: Provide persistence when satisfying storage policy.: Resolved
• [SPS]: Daemon thread of SPS should start only in Active NN: Resolved
• [SPS]: chooseTargetTypeInSameNode should pass accurate block size to chooseStorage4Block while choosing target: Resolved
• [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues: Resolved
• [SPS]: Check Mover file ID lease also to determine whether Mover is running: Resolved
• [SPS]: Remove xAttrs when movements done or SPS disabled: Resolved
• [SPS]: Fix timeout issue in unit tests caused by longger NN down time: Resolved
• [SPS]: NN switch and rescheduling movements can lead to have more than one coordinator for same file blocks: Resolved
• [SPS]: SPS should clean Xattrs when no blocks required to satisfy for a file: Resolved
• [SPS]: fix issue of moving blocks with satisfier while changing replication factor: Resolved
• [SPS]: Namenode failed to start while loading SPS xAttrs from the edits log.: Resolved
• [SPS] : Empty files should be ignored in StoragePolicySatisfier.: Resolved
• [SPS] : Handle NPE in BlockStorageMovementTracker when dropSPSWork() called: Resolved
• [SPS] : StoragePolicySatisfier should not select same storage type as source and destination in same datanode.: Resolved
• [SPS] Correct the log in BlockStorageMovementAttemptedItems#blockStorageMovementResultCheck: Resolved
• [SPS]: Add CLI command for satisfy storage policy operations: Resolved
• [SPS]: Should give chance to satisfy the low redundant blocks before removing the xattr: Resolved
• [SPS]: Double checks to ensure that SPS/Mover are not running together: Resolved
• [SPS]: Document the SPS feature: Resolved
• [SPS] : Fix TestStoragePolicySatisfierWithStripedFile#testSPSWhenFileHasLowRedundancyBlocks: Resolved
• [SPS]: Fix checkstyle warnings: Resolved
• [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to improve thread cleanup time: Resolved
• [SPS]: Fix review comments of StoragePolicySatisfier feature: Resolved
• [SPS]: Optimize extended attributes for tracking SPS movements: Resolved
• [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir: Resolved
• [SPS] : Block movement analysis should be done in read lock.: Resolved
• [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements: Resolved
• [SPS]: Provide an option to track the status of in progress requests: Resolved
• [SPS]: Improve storage policy satisfier configurations: Resolved
• [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and HDFS-11968 commits: Resolved
• [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations.: Resolved
• [SPS]: Move SPS classes to a separate package: Resolved
• [SPS]: Reduce the locking and cleanup the Namesystem access: Resolved
• [SPS]: Implement a mechanism to scan the files for external SPS: Resolved
• [SPS]: Implement a mechanism to do file block movements for external SPS: Resolved
• [SPS] : Create start/stop script to start external SPS process.: Resolved
• [SPS]: Revisit configurations to make SPS service modes internal/external/none: Resolved
• [SPS]: Provide External Context implementation.: Resolved
• [SPS]: Fix review comments of external storage policy satisfier: Resolved
• [SPS]: Fix the branch review comments(Part1): Resolved
• [SPS]: Reduce the number of APIs in NamenodeProtocol used by external satisfier: Resolved
• [SPS]: Collects successfully moved block details via IBR: Resolved
• [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls: Resolved
• [SPS]: Use DFSUtilClient#makePathFromFileId() to prepare satisfier file path: Resolved
• [SPS]: Cleanup work for HDFS-10285: Resolved
• [SPS]: Fix the branch review comments: Resolved
• [SPS] : Merge work for HDFS-10285 branch: Resolved
• [SPS]: Remove unwanted FSNamesystem #isFileOpenedForWrite() and #getFileInfo() function: Resolved
• [SPS]: Fix bug for unit test of reconfiguring SPS mode: Resolved
• [SPS]: Handle failure retries for moving tasks: Resolved
• [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS: Resolved
• [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread: Resolved
• [SPS]: allow re-satisfy path after restarting sps process: Open
• [SPS]: Expose metrics to JMX for external SPS: Resolved
• [SPS]: Should not start indefinitely while another SPS process is running: Resolved