[HDFS-7240] Scaling HDFS

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Assignee: Jitendra Nath Pandey
• Reporter: Jitendra Nath Pandey
• Created: 13/Oct/14 23:34
• Updated: 27/Apr/18 17:39

Description

HDFS Scalability-v2.pdf describes areas where HDFS does well and its scaling challenges and how to address those challenges. Scaling HDFS requires scaling the namespace layer and also the block layer. This jira provides a new block layer,  Hadoop Distributed Storage Layer (HDSL), that scales the block layer by grouping blocks into containers thereby reducing the block-to-location map and also reducing the number of block reports and their processing

A scalable namespace can be put on top this scalable block layer:

• HDFS-10419 describes how the existing NN can be modified to use the new block layer.
• HDFS-13074  also provides, as an interim step; a scalable flat Key-Value namespace  on top of the new block layer; while it does not provide the HDFS API, it does support the Hadoop FS APIs (Hadoop FileSystem, FileContext).

 

Old Description

This jira proposes to add object store capabilities into HDFS.
As part of the federation work (HDFS-1052) we separated block storage as a generic storage layer. Using the Block Pool abstraction, new kinds of namespaces can be built on top of the storage layer i.e. datanodes.
In this jira I will explore building an object store using the datanode storage, but independent of namespace metadata.

I will soon update with a detailed design document.

Issue Links

• is related to: HDDS-4 Implement security for Hadoop Distributed Storage Layer
• is required by: HDFS-10419 Building HDFS on top of new storage layer (HDDS)
• relates to: HDFS-11118 Block Storage for HDFS
• relates to: HDFS-13074 Ozone File System

Sub-Tasks

• DataNode support for multiple datasets: Resolved
• Ozone: Introduce STORAGE_CONTAINER_SERVICE as a new NodeType.: Resolved
• Ozone: Refactor FsDatasetSpi to pull up HDFS-agnostic functionality into parent interface: Closed
• OzoneHandler : Add Quota Support: Resolved
• Create REST Interface for Volumes: Resolved
• OzoneHandler : Add Error Table: Resolved
• OzoneHandler: Add userAuth Interface and Simple userAuth handler: Resolved
• Ozone: fix the missing entries in hdfs script: Resolved
• Ozone: TestBlockPoolManager fails in ozone branch.: Resolved
• Ozone:SCM: Add support for registerNode in datanode: Resolved
• OzoneHandler : Support List and Info Volumes: Resolved
• Ozone: StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.: Resolved
• Ozone: Add Ozone Client lib for volume handling: Resolved
• Ozone: Support starting StorageContainerManager as a daemon: Resolved
• Ozone : move Chunk IO and container protocol calls to hdfs-client: Resolved
• Ozone : fix XceiverClient slow shutdown: Resolved
• Ozone : move ozone XceiverClient to hdfs-client: Resolved
• Ozone: SCM: Add datanode protocol: Resolved
• Ozone: TestEndpoint task failure: Resolved
• Ozone:SCM: Add support for registerNode in SCM: Resolved
• Ozone : add client-facing container APIs and container references: Resolved
• OzoneHandler : Add volume handler: Resolved
• Ozone: SCM: Make SCM use container protocol: Resolved
• DataNode should filter the set of NameSpaceInfos passed to Datasets: Closed
• OzoneHandler : Add ACL support: Resolved
• Move DatasetSpi to new package: Closed
• OzoneHandler : Add Local StorageHandler support for volumes: Resolved
• Ozone: Introduce KeyValueContainerDatasetSpi: Closed
• OzoneHandler : Add Volume Interface to Data Node HTTP Server: Resolved
• OzoneHandler : Add common bucket objects: Resolved
• OzoneHandler: Integration of REST interface and container data pipeline back-end: Resolved
• OzoneHandler : Add Bucket REST Interface: Resolved
• Ozone: Add small file support RPC: Resolved
• Ozone: Cleanup some dependencies: Resolved
• Ozone: use containers with the state machine: Resolved
• Ozone: Add allocateContainer RPC: Resolved
• Ozone: Unify StorageContainerConfiguration with ozone-default.xml & ozone-site.xml: Resolved
• OzoneHandler : Enable stand-alone local testing mode: Resolved
• OzoneHandler : Add localStorageHandler support for Buckets: Resolved
• OzoneHandler : Enable MiniDFSCluster based testing for Ozone: Resolved
• Ozone: Add container definitions: Resolved
• Exclude Ozone protobuf-generated classes from Findbugs analysis.: Resolved
• Ozone: Add container transport server: Resolved
• Ozone: Add container transport client: Resolved
• Stop tracking CHANGES.txt in the HDFS-7240 feature branch.: Resolved
• OzoneHandler : Add Key handler: Resolved
• Ozone: Add Ozone Client lib for bucket handling: Resolved
• ozone : Add volume commands to CLI: Resolved
• Ozone : Add container dispatcher: Resolved
• Ozone: Add buckets commands to CLI: Resolved
• OzoneHandler : Add localstorage support for keys: Resolved
• Ozone: Adding logging support: Resolved
• Ozone: Refactor container Namespace: Resolved
• Ozone : Enable better error reporting for failed commands in ozone shell: Resolved
• ozone : Add key commands to CLI: Resolved
• Ozone: Add container persistence: Resolved
• Ozone: Implement storage container manager: Resolved
• Fix failing HDFS tests on HDFS-7240 Ozone branch.: Resolved
• Ozone : Add chunk persistance: Resolved
• Ozone: Add key Persistence: Resolved
• Ozone: end-to-end integration for create/get volumes, buckets and keys.: Resolved
• Ozone: Make config key naming consistent: Resolved
• Ozone: Shutdown cleanly: Resolved
• Ozone : exclude cblock protobuf files from findbugs check: Resolved
• Ozone : reuse Xceiver connection: Resolved
• Ozone : move StorageContainerLocation protocol to hdfs-client: Resolved
• Ozone: Add metrics for container operations and export over JMX: Resolved
• Ozone:SCM: Add chill mode support to NodeManager.: Resolved
• Ozone: Add paging support to list Volumes: Resolved
• Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.: Resolved
• Ozone: Introduce new config keys for SCM service endpoints: Resolved
• Fix Ozone unit tests to use MiniOzoneCluster: Resolved
• Ozone: SCM: Add NodeManager: Resolved
• Ozone: TestContainerMapping needs to cleanup levelDB files: Resolved
• Ozone: SCM: Send node report to SCM with heartbeat: Resolved
• Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate: Resolved
• Ozone: SCM: Add negative tests cases for datanodeStatemachine: Resolved
• Ozone: Exclude container protobuf files from findbugs check: Resolved
• Ozone: Fix Container test regression.: Resolved
• Ozone: TestDatanodeStateMachine is failing intermittently: Resolved
• Ozone: Add protobuf definitions for container reports: Resolved
• Ozone: Fix TestEndpoint test regression: Resolved
• Ozone: Improve logging and error handling in the container layer: Resolved
• Ozone:SCM: Support MXBean for SCM and NodeManager: Resolved
• Ozone: Separate XceiverServer and XceiverClient into interfaces and implementations: Resolved
• Ozone: SCM: Move SCM config keys to ScmConfig: Resolved
• Ozone: SCM: Container allocation based on node report: Resolved
• Ozone: Fix TestContainerPersistence failures: Resolved
• Ozone: Fix TestDatanodeStateMachine#testDatanodeStateContext: Resolved
• Ozone: SCM: Add close container RPC: Resolved
• Ozone: Fix datanode ID handling in MiniOzoneCluster: Resolved
• Ozone: Add the ability to handle sendContainerReport Command: Resolved
• Ozone: Merge with trunk needed a ignore duplicate entry in pom file due to shading: Resolved
• Ozone: MiniOzoneCluster prints too many log messages by default: Resolved
• Ozone: Add a check to prevent removing a container that has keys in it: Resolved
• Ozone: SCM: Add node pool management API: Resolved
• Ozone: SCM: Support update container: Resolved
• Ozone: close container should call compactDB: Resolved
• Ozone: SCM: Add Comparable Metric Support: Resolved
• Ozone: Implement XceiverServerSpi and XceiverClientSpi using Ratis: Resolved
• Ozone: Document missing metrics for container operations: Resolved
• Ozone: Unit tests running failed in Windows: Resolved
• Ozone: Allocate container for MiniOzone cluster fails because of insufficient space error: Resolved
• Ozone: Fix UT failures that caused by hard coded datanode data dirs: Resolved
• Ozone: support setting chunk size in streaming API: Resolved
• Ozone:SCM: Remove null command: Resolved
• Ozone: TestContainerPlacement fails because of string mismatch in expected Message: Resolved
• Ozone: Implement listKey function for KeyManager: Resolved
• Ozone: SCM: CLI: Design SCM Command line interface: Resolved
• Ozone: Add unit test for storage container metrics: Resolved
• Ozone: Support force delete a container: Resolved
• Ozone: SCM daemon is unable to be started via CLI: Resolved
• Ozone: SCM: CLI: Add shell code placeholder classes: Resolved
• Ozone: SCM: Add Block APIs: Resolved
• Ozone: Fix compile error due to inconsistent package name: Resolved
• Ozone: misc improvements for SCM CLI: Resolved
• Ozone: SCM: Support listContainers API: Resolved
• Ozone: SCM CLI: Implement delete container command: Resolved
• Ozone: add the DB names to OzoneConsts: Resolved
• Ozone: Revise create container CLI specification and implementation: Resolved
• Ozone:SCM: Add support for getContainer in SCM: Resolved
• Ozone : need to fix OZONE_SCM_DEFAULT_PORT: Resolved
• Ozone: Reuse ObjectMapper instance to improve the performance: Resolved
• Ozone: SCM: CLI: Add Debug command: Resolved
• Ozone : SCMNodeManager#close() should also close node pool manager object: Resolved
• Ozone: Get container report should only report closed containers: Resolved
• Ozone: SCM: CLI: Revisit delete container API: Resolved
• Ozone : add DEBUG CLI support of blockDB file: Resolved
• Ozone: Add finer error codes for container operaions: Resolved
• Ozone: SCM CLI: Implement info container command: Resolved
• Ozone: Fix spotbugs warnings: Resolved
• Ozone: Correct description for ozone.handler.type in ozone-default.xml: Resolved
• Ozone : add DEBUG CLI support for nodepool db file: Resolved
• Ozone: KSM: Create Key Space manager service: Resolved
• Ozone : add DEBUG CLI support for open container db file: Resolved
• Ozone: SCM: Support Delete Block: Resolved
• Ensure LevelDB DBIterator is closed: Resolved
• CBlockManager#main should join() after start() service: Resolved
• Ozone: Add archive support to containers: Resolved
• Ozone: fix the consistently timeout test testUpgradeFromRel22Image: Resolved
• Ozone: Improve the way of getting test file path in unit tests for Windows: Resolved
• Ozone: Fix javac warnings: Resolved
• Ozone : add sql debug CLI to hdfs script: Resolved
• Ozone: XceiverClientRatis should implement XceiverClientSpi.connect(): Resolved
• Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes does not remove chosen node from healthy list.: Resolved
• Ozone: Fix Http connection leaks in ozone clients: Resolved
• Ozone: Stack Overflow in XceiverClientManager because of race condition in accessing openClient: Resolved
• Ozone: Datanode needs to re-register to SCM if SCM is restarted: Resolved
• Ozone : need to refactor StorageContainerLocationProtocolServerSideTranslatorPB: Resolved
• Ozone : implement StorageContainerManager#getStorageContainerLocations: Resolved
• Ozone: KSM: Add createVolume API: Resolved
• Ozone: KSM: Add setVolumeProperty: Resolved
• Ozone: KSM : add createBucket: Resolved
• Ozone: KSM: add getBucketInfo: Resolved
• Ozone: SCM: Separate BlockLocationProtocol from ContainerLocationProtocol: Resolved
• Ozone: KSM : Add putKey: Resolved
• Ozone: Do not initialize Ratis cluster during datanode startup: Resolved
• Ozone: CLI: Guarantees user runs SCM commands has appropriate permission: Resolved
• Ozone: KSM: Add getKey: Resolved
• Ozone: KSM: add SetBucketProperty: Resolved
• Ozone:KSM : add deleteVolume: Resolved
• Ozone: Differentiate time interval for different DatanodeStateMachine state tasks: Resolved
• Ozone: Cleaning up local storage when closing MiniOzoneCluster: Resolved
• Ozone: Cleanup imports: Resolved
• Ozone: Add Ratis management API: Resolved
• Ozone: TestKeySpaceManager#testDeleteVolume fails: Resolved
• Ozone: MiniOzoneCluster should set "ozone.handler.type" key correctly: Resolved
• Ozone: KSM: Add checkVolumeAccess: Resolved
• Ozone: KSM: Add deleteKey: Resolved
• Ozone: SCM: TestNodeManager takes too long to execute: Resolved
• Shared XceiverClient should be closed if there is no open clients to avoid resource leak: Resolved
• Ozone: Create metadata path automatically after null checking: Resolved
• Ozone: Implement a common helper to return a range of KVs in levelDB: Resolved
• Ozone: KSM: add deleteBucket: Resolved
• Ozone: KSM: Remove protobuf formats from KSM wrappers: Resolved
• Ozone: KSM: add listBuckets: Resolved
• Ozone: Fix TestContainerSQLCli#testConvertContainerDB: Resolved
• Remove Guava v21 usage from HDFS-7240: Resolved
• Ozone: remove disabled tests: Resolved
• Ozone: Output error when DN handshakes with SCM: Resolved
• Ozone: Ensure KSM is initiated using ProtobufRpcEngine: Resolved
• Ozone: Containers in different datanodes are mapped to the same location: Resolved
• Ozone: Add start-ozone.sh to quickly start ozone.: Resolved
• Ozone: Add stop-ozone.sh script: Resolved
• Ozone: cannot enable test debug/trace log: Resolved
• Ozone: TestContainerPersistence never uses MiniOzoneCluster: Resolved
• Ozone: KSM: Add listKey: Resolved
• Ozone: Support force update a container: Resolved
• Ozone: Clarify startup error message of Datanode in case namenode is missing: Resolved
• Ozone: Documentation: Add getting started page: Resolved
• Ozone: Add documentation for metrics in KSMMetrics to OzoneMetrics.md: Resolved
• Ozone: TestXceiverClientManager.testFreeByEviction fails occasionally: Resolved
• Ozone: SCM: Container metadata are not loaded properly after datanode restart: Resolved
• Ozone: Recover SCM state when SCM is restarted: Resolved
• Ozone: Add all configurable entries into ozone-default.xml: Resolved
• Ozone: Encapsulate KSM metadata key for better (de)serialization: Resolved
• Ozone: Enable HttpServer2 for SCM and KSM: Resolved
• Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml: Resolved
• Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.: Resolved
• Ozone: Rename OzoneClient to OzoneRestClient: Resolved
• Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown: Resolved
• Ozone: Ozone shell: Multiple RPC calls for put/get key: Resolved
• Ozone: KSM : add listVolumes: Resolved
• Ozone: Set proper parameter default values for listBuckets http request: Resolved
• Ozone : SCM cli misc fixes/improvements: Resolved
• Ozone: CLI: support infoKey command: Resolved
• Ozone: Test if all the configuration keys are documented in ozone-defaults.xml: Resolved
• Ozone: ozone server should create missing metadata directory if it has permission to: Resolved
• Ozone: Add REST API documentation: Resolved
• Ozone:TestOzoneContainerRatis & TestRatisManager are failing consistently: Resolved
• Ozone: listKey doesn't work from ozone commandline: Resolved
• Ozone: Add infoKey REST API document: Resolved
• Ozone: Add the unit test for KSMMetrics: Resolved
• Ozone: Review all cases where we are returning FAILED_INTERNAL_ERROR: Resolved
• Ozone: OzoneClient: Implementation of OzoneClient: Resolved
• Ozone: SCM CLI: Implement list container command: Resolved
• Ozone: Improvement rest API output format for better looking: Resolved
• Ozone: Fix UT failure in TestOzoneConfigurationFields: Resolved
• Ozone: listVolumes doesn't work from ozone commandline: Resolved
• Ozone: Bucket versioning design document: Resolved
• Ozone: SCM http server is not stopped with SCM#stop(): Resolved
• Ozone: KSM: previous key has to be excluded from result in listVolumes, listBuckets and listKeys: Resolved
• Ozone: SCM: Add the ability to handle container reports: Resolved
• Ozone: OzoneClient: Abstraction of OzoneClient and default implementation: Resolved
• Ozone: add TestKeysRatis, TestBucketsRatis and TestVolumeRatis: Resolved
• Ozone: KSM: Cleanup of keys in KSM for failed clients: Resolved
• Ozone: Create a general abstraction for metadata store: Resolved
• Ozone: TestOzoneConfigurationFields is failing because ozone-default.xml has some missing properties: Resolved
• Ozone: Ozone shell: Add more testing for volume shell commands: Resolved
• Ozone: OzoneClient: OzoneClientImpl add getDetails calls: Resolved
• Ozone : add an UT to test partial read of chunks: Resolved
• Ozone: Ozone shell: Add more testing for bucket shell commands: Resolved
• Ozone: Corona: Implementation of Corona: Resolved
• Ozone: RocksDB implementation of ozone metadata store: Resolved
• Ozone: Ozone shell: Add more testing for key shell commands: Resolved
• Ozone: TestNodeManager times out before it is able to find all nodes: Resolved
• Ozone: Fix TestContainerReplicationManager by setting proper log level for LogCapturer: Resolved
• Ozone: Fix Leaking in TestStorageContainerManager#testRpcPermission: Resolved
• Ozone: Fix Leaking in TestXceiverClientManager: Resolved
• Ozone: SCM: Add queryNode RPC Call: Resolved
• Ozone: OzoneFileSystem: Ozone & KSM should support "/" delimited key names: Resolved
• Ozone: OzoneFileSystem: KSM should maintain key creation time and modification time: Resolved
• Ozone: Corona: Add stats and progress bar to corona: Resolved
• Ozone : add RocksDB support to DEBUG CLI: Resolved
• Ozone: Intermittent failure TestContainerPersistence#testListKey: Resolved
• Ozone: Fix the remaining failure tests for Windows caused by incorrect path generated: Resolved
• Ozone : add support to DEBUG CLI for ksm.db: Resolved
• Ozone: OzoneClient: OzoneClientImpl Add setBucketProperty and delete calls: Resolved
• Ozone: KSM : Use proper defaults for block client address: Resolved
• Ozone: Corona: Adding corona as part of hdfs command: Resolved
• Ozone : add key partition: Resolved
• Ozone: DeleteKey-1: KSM replies delete key request asynchronously: Resolved
• Ozone: Web interface for KSM: Resolved
• Ozone: Ensures listKey lists all required key fields: Resolved
• OzoneClientUtils#updateListenAddress should use server address to update listening addrees: Resolved
• Ozone: Support asynchronus client API for SCM and containers: Resolved
• Ozone: TestStorageContainerManager#testRpcPermission fails when ipv6 address used: Resolved
• Ozone: OzoneClient: Handling SCM container creationFlag at client side: Resolved
• Ozone: KeySpaceManager should unregister KSMMetrics upon stop: Resolved
• Ozone: Reduce MiniOzoneCluster handler thread count: Resolved
• Ozone: potential thread leaks: Resolved
• Ozone: KSM: Add creation time field in volume info: Resolved
• Ozone: KSM: Reduce default handler thread count from 200: Resolved
• Ozone: OzoneClient: Refactor and move ozone client from hadoop-hdfs to hadoop-hdfs-client: Resolved
• Ozone: XceiverClientManager should not close the connection if client holds the reference: Resolved
• Ozone : add debug cli to hdfs script: Resolved
• Ozone: Corona: move corona from test to tools package: Resolved
• Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks at background: Resolved
• Ozone: Extend MBeans utility to add any key value pairs to the registered MXBeans: Resolved
• Ozone: Ozone-default.xml has 3 properties that do not match the default Config value: Resolved
• Ozone: Web interface for SCM: Resolved
• Ozone: KSM: Add creation time field in bucket info: Resolved
• Ozone: Block deletion service floods the log when deleting large number of block files: Resolved
• Ozone: Add valid trace ID check in sendCommandAsync: Resolved
• Ozone: SCM: Add StateMachine for pipeline/container: Resolved
• Ozone: TestKeys#testPutAndGetKeyWithDnRestart fails: Resolved
• Ozone: SCM: Add create replication pipeline RPC: Resolved
• Ozone: Implement update volume owner in ozone shell: Resolved
• Ozone: SCM: move container/pipeline StateMachine to the right package: Resolved
• Ozone: TestKeys is failing consistently: Resolved
• Ozone: support setting timeout in background service: Resolved
• Ozone: the static cache provided by ContainerCache does not work in Unit tests: Resolved
• Ozone: Concurrent RocksDB open calls fail because of "No locks available": Resolved
• Ozone: DeleteKey-5: Implement SCM DeletedBlockLog: Resolved
• Ozone: SCM: use state machine for open containers allocated for key/blocks: Resolved
• Ozone: TestQueryNode#testStaleNodesCount is failing.: Resolved
• Ozone: TestOzoneContainer#testCreateOzoneContainer fails: Resolved
• TestKeys#testPutAndGetKeyWithDnRestart failed: Resolved
• Ozone: maven dist compilation fails with "Duplicate classes found" error: Resolved
• Ozone: Corona: Support for random validation of writes: Resolved
• Ozone: DeleteKey-4: Block delete between SCM and DN: Resolved
• Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient: Resolved
• Ozone: Shuffle container list for datanode BlockDeletingService: Resolved
• Ozone: SCM failed to start when a container metadata is empty: Resolved
• Ozone: Refactor KSM metadata class names to avoid confusion: Resolved
• Ozone: Fix TestArchive#testArchive: Resolved
• Ozone: Extend Datanode web interface with SCM information: Resolved
• Ozone: Too many open files error while running corona: Resolved
• Ozone: OzoneFileSystem: OzoneFileystem initialization code: Resolved
• Ozone: TestKSMSQLCli is not working as expected: Resolved
• Ozone: SCM: BlockManager creates a new container for each allocateBlock call: Resolved
• Ozone: Storage container data pipeline: Resolved
• Ozone: Datanode is unable to register with scm if scm starts later: Resolved
• Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions: Resolved
• Ozone: Implement TopN container choosing policy for BlockDeletionService: Resolved
• Ozone: oz commandline list calls should return valid JSON format output: Resolved
• Ozone: KSM: Garbage collect deleted blocks: Resolved
• Ozone: OzoneClient: Refactoring OzoneClient API: Resolved
• Ozone: KSM: Unable to put keys with zero length: Resolved
• Add markdown documentation about Ozone: Resolved
• Ozone: Object store handler supports reusing http client for multiple requests.: Resolved
• Ozone: Cleanup Checkstyle issues: Resolved
• Ozone: TopN container choosing policy should ignore containers that has no pending deletion blocks: Resolved
• Ozone: SCM CLI: Implement close container command: Resolved
• Ozone: KSM: set creationTime for volume/bucket/key: Resolved
• Ozone: Implement the trace ID generator: Resolved
• Ozone: OzoneClient : Remove createContainer handling from client: Resolved
• Ozone: SCM: Container State Machine -1- Track container creation state in SCM: Resolved
• Ozone: KSM: multiple delete methods in KSMMetadataManager: Resolved
• Ozone: Corona: Support for variable key length in offline mode: Resolved
• Ozone: fix a KeySpaceManager startup message typo: Resolved
• Ozone: fix hard coded version in the Ozone GettingStarted guide: Resolved
• Ozone: Fix TestXceiverClientMetrics#testMetrics: Resolved
• Ozone: TestAllocateContainer fails on jenkins: Resolved
• Ozone: Misc : Make sure that ozone-site.xml is in etc/hadoop in tarball created from mvn package.: Resolved
• Ozone: Ozone data placement is not even: Resolved
• Ozone: Add metrics for pending storage container requests: Resolved
• Ozone: Ratis: Readonly calls in XceiverClientRatis should use sendReadOnly: Resolved
• Ozone : the sample ozone-site.xml in OzoneGettingStarted does not work: Resolved
• Ozone: SCM: NodeManager should log when it comes out of chill mode: Resolved
• Ozone : handle inactive containers on DataNode: Resolved
• Ozone: Container: Move IPC port to 98xx range: Resolved
• Ozone: Support Ratis as a first class replication mechanism: Resolved
• Ozone: OzoneFileSystem: read/write/create/open/getFileInfo APIs: Resolved
• Ozone: List Key on an empty ozone bucket fails with command failed error: Resolved
• Ozone: More detailed documentation about the ozone components: Resolved
• Ozone: OzoneClient: Add list calls in RpcClient: Resolved
• OzoneFileSystem: A Hadoop file system implementation for Ozone: Resolved
• Ozone: enforce DependencyConvergence uniqueVersions: Resolved
• Ozone: KSM: Make "ozone.ksm.address" as mandatory property for client: Resolved
• Ozone: KSM: Changing log level for client calls in KSM: Resolved
• Ozone: OzoneRestClientException swallows exceptions which makes client hard to debug failures: Resolved
• Ozone: Cleanup findbugs issues: Resolved
• Ozone: Cleanup javadoc issues: Resolved
• Ozone: OzoneRestClient timeout is not configurable: Resolved
• Ozone: Cleanup javac issues: Resolved
• Ozone: some UX improvements to oz_debug: Resolved
• Ozone: Revert files not related to ozone change in HDFS-7240 branch: Resolved
• Ozone: mvn package compilation fails on HDFS-7240: Resolved
• Ozone: mvn package is failing with out skipshade: Resolved
• Ozone: ListBucket is too slow: Resolved
• Ozone: Create UI page to show Ozone configs by tags: Resolved
• Ozone: DB listing operation performance improvement: Resolved
• Ozone: Add tags to config: Closed
• Ozone: Record number of keys scanned and hinted for getRangeKVs call: Resolved
• Ozone: OzoneClient: Verify bucket/volume name in create calls: Resolved
• javadoc: error - class file for org.apache.http.annotation.ThreadSafe not found: Resolved
• Ozone: Reduce key creation overhead in Corona: Resolved
• Ozone: node status text reported by SCM is a bit confusing: Resolved
• Ozone: Create docker-compose definition to easily test real clusters: Resolved
• Ozone: refactor some functions in KSMMetadataManagerImpl to be more readable and reusable: Resolved
• Ozone: Improve the formatting of the RPC stats on web UI: Resolved
• Ozone: Add ASF license to docker files: Resolved
• Ozone : allow create key without specifying size: Resolved
• Ozone: Some minor text improvement in SCM web UI: Resolved
• Ozone: Fix TestDatanodeStateMachine#testDatanodeStateMachineWithInvalidConfiguration: Resolved
• Ozone: TestBlockDeletingService#testBlockDeletionTimeout sometimes timeout: Resolved
• Ozone: Documentation: Add Ozone overview documentation.: Resolved
• Ozone: SCM: Open containers are not reused for block allocation after restart: Resolved
• Ozone: Creation time wrongly showed in OzoneRest.md: Resolved
• Ozone: update Ratis to the latest snapshot: Resolved
• Ozone: Cleanup the ozone-default.xml: Resolved
• Ozone: Reduce some KSM/SCM deletion log messages from INFO to DEBUG: Resolved
• Add Ozone config UI to KSM and SCM index page: Resolved
• Ozone: Fix swallow exceptions which makes hard to debug failures: Resolved
• Ozone: ContainerStorageLocation#scmUsage should count only SCM usage: Resolved
• Ozone: SCM Cli: Use the hostname of SCM server instead of the bind address: Resolved
• Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list: Resolved
• Ozone: SCM: Refactoring of chill mode logic in NodeManager: Resolved
• Ozone: remove spaces from the beginning of the hdfs script: Resolved
• Ozone: Fix 3 node ratis replication in Ozone: Resolved
• Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs: Resolved
• Ozone: Remove Warnings when building: Resolved
• Ozone : add documentation about Datanode http address: Resolved
• Ozone: Lease Manager framework: Resolved
• Ozone: Mini cluster can't start up on Windows after HDFS-12159: Resolved
• Ozone: OzoneClient: Add creation/modification time information in OzoneVolume/OzoneBucket/OzoneKey: Resolved
• Ozone: Add container usage information to DN container report: Resolved
• Ozone: TestXceiverClientManager and TestAllocateContainer occasionally fails: Resolved
• Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem: Resolved
• Ozone: Support optional documentation link in KSM/SCM webui: Resolved
• Ozone: KSM UI: Add description for configs in UI: Resolved
• Ozone: Log elapsed time for block deletion tasks: Resolved
• Ozone: dozone: Use (proposed) base image from HADOOP-14898: Resolved
• Ozone: SCM: Lease support for container creation: Resolved
• Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui: Resolved
• Ozone: Add hugo to the dev docker image: Resolved
• Ozone: SCM: Add Node Metrics for SCM: Resolved
• Ozone: SCM should read all Container info into memory when booting up: Resolved
• Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket: Resolved
• Ozone: use specific names for RPC.Server instances: Resolved
• Ozone: SCMMXBean NodeCount is overlapping with NodeManagerMXBean: Resolved
• Ozone: SCM: Clean up of container in DELETING state: Resolved
• Ozone: generate swagger descriptor for the Ozone REST Api: Resolved
• Ozone: Decrease interval time of SCMBlockDeletingService for improving the efficiency: Resolved
• Ozone: Enable XFrame options for KSM/SCM web ui: Resolved
• Ozone: TestLeaseManager#testLeaseCallbackWithMultipleLeases fails: Resolved
• Ozone: fix javadoc/unit test for calling MetadataStore.getRangeKVs with non existing key: Resolved
• Ozone: Corona: Add statistical information to json output: Resolved
• Ozone: Fix Ozone's env vars (again): Resolved
• Ozone: SCM: Handling container report with key count and container usage.: Resolved
• Ozone: Improve SCM block deletion throttling algorithm: Resolved
• Add Support for SCM --init command: Resolved
• Use GenericOptionsParser for scm and ksm daemon: Resolved
• Ozone: Fix inconsistent default value in ozone settings: Resolved
• Ozone: add logger for oz shell commands and move error stack traces to DEBUG level: Resolved
• Ozone services must stay disabled in secure setup for alpha: Resolved
• Ozone: Add timeunit for ozone.scm.heartbeat.interval.seconds in ozone-default.xml: Resolved
• Ozone: add port/service names to the ksm/scm web ui: Resolved
• Ozone: scm web: fix the node status table: Resolved
• Ozone : TestSCMCli is failing consistently: Resolved
• Ozone: KSM: ADD support for KSM --createObjectStore command: Resolved
• Ozone: SCM: Aggregate the metrics from all the container reports: Resolved
• Ozone: TestOzoneConfigurationFields fails consistently: Resolved
• Ozone: SCM: BlockManagerImpl close is called twice during StorageContainerManager#stop: Resolved
• Ozone: Expose RockDB stats via JMX for Ozone metadata stores: Resolved
• Ozone: Turn off full synced write in RocksDB MetadataStore: Resolved
• Ozone: TestOzoneRpcClient#testPutKeyRatisThreeNodes fails: Resolved
• Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED: Resolved
• Ozone: TestContainerPersistence#testListContainer fails frequently due to timed out: Resolved
• Ozone: moving NodeType from OzoneConsts to Ozone.proto: Resolved
• Ozone: Upgrade to latest ratis build: Resolved
• Ozone: SCM: Expose StorageContainerManager#getScmId through container protocol: Resolved
• Ozone: OzoneClient: Support for REST protocol: Resolved
• Ozone : delete open key entries that will no longer be closed: Resolved
• Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.: Resolved
• Ozone: Container : Add key versioning support-1: Resolved
• Ozone: Add datanodeID to heartbeat responses and container protocol: Resolved
• Ozone: Use time units in the Ozone configuration values: Resolved
• Ozone: TestSCMCli#testHelp and TestSCMCli#testNonExistCommand fail consistently: Resolved
• Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto: Resolved
• Ozone: SCM: Support for Container LifeCycleState PENDING_CLOSE and LifeCycleEvent FULL_CONTAINER: Resolved
• Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close: Resolved
• Ozone : better handling of operation fail due to chill mode: Resolved
• Ozone: Fix TestStorageContainerManager#testBlockDeletionTransactions: Resolved
• Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client: Resolved
• Ozone: Redundant characters printed in exception log: Resolved
• Ozone: web: integrate configuration reader page to the SCM/KSM web ui.: Resolved
• Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch: Resolved
• Ozone: XceiverClient should have upper bound on async requests: Resolved
• SCM should not start if Cluster Version file does not exist: Resolved
• Ozone: write chunk call fails because of Metrics registry exception: Resolved
• Ozone: Optimize chunk writes for Ratis by avoiding double writes: Resolved
• Ozone: Service Discovery: RPC endpoint in KSM for getServiceList: Resolved
• Ozone: ConfServlet does not trim values during the description parsing: Resolved
• SCM should support a RPC to share the cluster Id with KSM and DataNodes: Resolved
• Ozone: Fix TestKeySpaceManager#testKSMInitializationFailure: Resolved
• Ozone: TestContainerPersistence is failing becaue of container data mismatch: Resolved
• Ozone: Client: TestOzoneRpcClient#testPutKeyRatisThreeNodes is failing: Resolved
• Ozone : TestSCMCli and TestContainerStateManager tests are failing consistently while updating the container state info.: Resolved
• Ozone: remove setAllocatedBytes method in ContainerInfo: Resolved
• Ozone: Service Discovery: REST endpoint in KSM for getServiceList: Resolved
• Ozone: Use relative resource urls in KSM/SCM web ui: Resolved
• Ozone: owner name should be set properly when the container allocation happens: Resolved
• Ozone: SCM: Restructuring container state transition and events: Resolved
• Ozone: Service Discovery: Adding REST server details in ServiceList: Resolved
• Fix closeContainer API with the right container state change: Resolved
• Ozone: Avoid configuration.get() call for each container report: Resolved
• Ozone : add scm init command to document.: Resolved
• TestOzoneConfigurationFields fails due to missing configs in ozone-default.xml: Resolved
• Ozone: ContainerStateMachine should synchronize operations between createContainer and writeChunk: Resolved
• Ozone: DatanodeStateMachine: Handling Uncaught Exception in command handler thread: Resolved
• Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally: Resolved
• Ozone : Add an API to get Open Container by Owner, Replication Type and Replication Count: Resolved
• Ozone: dozone: initialize scm and ksm directory on cluster startup: Resolved
• Ozone: Service Discovery API: Resolved
• Ozone: Remove the Priority Queues used in the Container State Manager: Resolved
• Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-0f7169d-SNAPSHOT): Resolved
• Ozone: XceiverClientManager should cache objects based on pipeline name: Patch Available
• Ozone: Improve SQLCLI performance: Patch Available
• Ozone: Documentation: Add version specific documentation site to KSM: Resolved
• Ozone: Container : Add key versioning support-2: Patch Available
• Ozone: Non-admin user is unable to run InfoVolume to the volume owned by itself: In Progress
• Ozone: KSM: Allocate key should honour volume quota if quota is set on the volume: In Progress
• Ozone : debug cli: add support to load user-provided SQL query: In Progress
• Ozone: Container : Add key versioning support-3: In Progress
• Ozone : add read/write random access to Chunks of a key: In Progress
• Ozone: Container : Add key versioning support-4: In Progress
• Ozone: Parallelize ChunkOutputSream Writes to container: Patch Available
• Make ContainerStateMachine#applyTransaction async: Resolved
• Ozone: TestContainerPersistence#testListContainer sometimes timeout: Resolved
• Ozone: generate optional, version specific documentation during the build: Resolved
• Ozone: SCM: avoid synchronously loading all the keys from containers upon SCM datanode start: Open
• Ozone: Ratis: Moving Ratis pipeline creation to client: Open
• Ozone: write deleted block to RAFT log for consensus on datanodes: Open
• Ozone: Documentation: Add Ozone-defaults documentation: Open
• Ozone: Purge metadata of deleted blocks after max retry times: Open
• Ozone: SCM CLI: Implement get container metrics command: Open
• Ozone: fix and reenable TestKeysRatis#testPutAndGetKeyWithDnRestart: Open
• Ozone: Support range in getKey operation: Open
• Ozone: Fix the Cluster ID generation code in SCM: Resolved
• Ozone:SCM: explore if we need 3 maps for tracking the state of nodes: Open
• Ozone: SCM: Handle duplicate Datanode ID: Resolved
• Ozone: add TestDistributedOzoneVolumesRatis, TestOzoneRestWithMiniClusterRatis and TestOzoneWebAccessRatis: Resolved
• Ozone: SCM: clean up containers that timeout during creation: Open
• Ozone: In Ratis, leader should validate ContainerCommandRequestProto before propagating it to followers: Resolved
• Ozone: Ozone shell: the root is assumed to hdfs: Open
• Ozone: change TestRatisManager to check cluster with data: Resolved
• Ozone: Container server needs enhancements to control of bind address for greater flexibility and testability.: Resolved
• Ozone:SCM: Add support for close containers in SCM: Resolved
• Ozone: Refactor Pipeline to have transport and container specific information: Resolved
• Ozone : Optimize putKey operation to be async and consensus: Open
• Ozone: provide a way to validate ContainerCommandRequestProto: Resolved
• ChunkManager functions do not use the argument keyName: Open
• Ozone: Replace Jersey container with Netty Container: Resolved
• Ozone: Document ozone metadata directory structure: Open
• OZone: SCM CLI: Implement get container command: Open
• Ozone: Compact DB should be called on Open Containers.: Open
• Ozone: Support SCM multiple instance for HA: Open
• Ozone:KSM: Add setVolumeAcls to allow adding/removing acls from a KSM volume: Resolved
• Ozone: KSM : Support for simulated file system operations: Open
• Ozone: Handle potential inconsistent states while listing keys: Open
• Ozone: Corona: Support for online mode: Resolved
• Ozone: Add client Block cache for SCM: Open
• Ozone: Put key operation concurrent executes failed on Windows: Open
• Ozone: SCM: Lease support for pipeline creation: Open
• Ozone: OzoneFileSystem: Add StorageStatistics to OzoneFileSystem: Open
• Ozone: start-all script is missing ozone start: Resolved
• Ozone: SCM: Make container report processing async: Open
• Ozone: SCM: ContainerStateManager#updateContainerState updates incorrect AllocatedBytes to container info.: Resolved
• Ozone: Documentation : Add ksm -createObjectStore command documentation.: Resolved
• Ozone: Service Discovery: Limit the number of datanodes (REST servers) returned by KeySpaceManager#getServiceList call: Open
• Ozone: Add support for Larger key sizes in Corona: Open
• Ozone: Make finalhash in ContainerInfo of StorageContainerDatanodeProtocol.proto optional: Resolved
• Ozone: OzoneFileSystem: Implement seek functionality for rest client: Open
• Ozone: TestContainerPersistence#testMultipleWriteSingleRead and TestContainerPersistence#testOverWrite fail consistently: Resolved
• Ozone: Add DN container open/close state to container report: Resolved
• Ozone: OzoneClient: Add configuration parameter to choose RestServerSelector implementation which will be used by RestClient: Open
• Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read timeout: Resolved
• Ozone: Update the allocatedBlock size in SCM when delete blocks happen.: Resolved
• Ozone: SCM: Support for container replica reconciliation - 1: Resolved
• Ozone: JMX name of RocksDbMetastore should be sanitized: Patch Available
• Ozone: Handle datanode failures in Storage Container Manager: Patch Available
• Ozone: Format open containers on datanode restart: Patch Available
• Ozone: DeleteContainer call in SCM does not remove the container info from the container inmemory maps.: Open
• Ozone: Rename Corona to Freon: Resolved
• Ozone: Make hadoop-common ozone free: Resolved
• Ozone: restructure Hdsl/Ozone code to separated maven subprojects: Resolved
• Ozone: Make ozone/hdsl/cblock modules turned off by default: Resolved
• Ozone: Remove DatanodeID dependency from HDSL and Ozone: Resolved
• Ozone: Remove containerPort, ratisPort and ozoneRestPort from DatanodeID and DatanodeIDProto: Resolved
• Ozone: fix classpath of yarn components: Resolved
• Document: update the new ozone docker file location: Resolved
• Ozone: Improve error message in case of missing nodes: Resolved
• Ozone: Update the ksm/scm CLI usage info: Resolved
• Ozone: docker-compose should only be copied to hadoop-dist if Phdsl is enabled.: Resolved
• Ozone: start-ozone.sh/stop-ozone.sh fail because of unresolved path: Resolved
• Ozone:Support for MicrobenchMarking Tool: Resolved
• Ozone: Remove InfoPort and InfoSecurePort from DatanodeDetails: Resolved
• Ozone: update log4j.properties changes for hdsl/ozone.: Resolved
• Ozone: Introduce a new SCM Exception which will be thrown when mandatory property is missing: Patch Available
• Ozone: Store only datanode UUID in datanode.id file: Open
• Ozone: remove hdsl/cblock/ozone source code from the official apache source release artifact: Resolved
• Ozone: Fix false positive RAT warning when project built without hds/cblock: Resolved
• Ozone: Move ozone specific ServiceRuntimeInfo utility to the framework: Resolved
• Ozone: Provide docker based acceptance testing on pseudo cluster: Resolved
• Ozone: Update IP and hostname in Datanode from SCM's response to the register call: Resolved
• Add config for min percentage of data nodes to come out of chill mode in SCM: Resolved
• Ozone: Remove commands from command queue when the datanode is declared dead: Open
• Ozone: The configuration of implement of DataNodeServicePlugin should obtain from datanode instance: Resolved
• Ozone: Make dependency of internal sub-module scope as provided in maven.: Resolved
• Ozone: ContainerID has incorrect package name: Resolved
• Ozone: Plugins support in HDSL Datanode Service: Resolved
• Ozone: Rename and fix ozone CLI scripts: Resolved
• Ozone: Rename HDSL to HDDS: Resolved
• Ozone: Use separated version schema for Hdds/Ozone projects: Resolved
• Ozone: Update existing Ozone documentation according to the recent changes: Resolved
• Ozone: Remove cblock code from HDFS-7240: Resolved
• Ozone: TestNodeManager tests fail: Resolved
• Ozone: Fix whitespaces and license issues in HDFS-7240 branch: Resolved
• Ozone: Clean-up of ozone related change from hadoop-hdfs-project: Resolved
• Ozone: Refactor MiniOzoneClassicCluster: Resolved
• Ozone: Clean-up of ozone related change from hadoop-common-project: Resolved
• Ozone: ClusterId and DatanodeUuid should be marked mandatory fields in SCMRegisteredCmdResponseProto: Resolved
• Ozone: Fix ConfServlet#getOzoneTags cmd: Resolved
• Ozone : Make BlockId in SCM a long value: Resolved
• Ozone: Handle Datanode Registration failure: Patch Available
• Ozone: Fix checkstyle issues in HDFS-7240: Resolved
• Ozone: TestBlockDeletingService test-case is failing: Patch Available
• Ozone: Clean-up of ozone related change from MiniDFSCluster: Resolved
• Httpserver start failure should be non fatal for KSM and SCM startup: Resolved
• Create IO provider for hdsl: Resolved
• Ozone: Fix precommit hook for Ozone/Hdds on trunk: Resolved
• Ozone: BlockManager MBean unregister failure upon shutdown: Resolved