Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11118

Block Storage for HDFS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • hdfs
    • None

    Description

      This JIRA proposes extending HDFS to provide replicated block storage capabilities using Storage Containers. This is would allow users to run unmodified programs that assume that they are running on a posix file system.

      With this extension, HDFS can be used like a block store. For example, YARN jobs could mount and use a volume at will. This is made possible by leveraging Storage Containers and will share the storage layer with Ozone and HDFS in future.

      Please see the attached design document for more details on this proposal.

      Attachments

        1. cblock-proposal.pdf
          306 kB
          Anu Engineer

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. HDFS-7240 Scaling HDFS

          • Major - Major loss of function.
          • Patch Available
          1.
          		 Block Storage : add block storage service protocol Sub-task Resolved Chen Liang
          2.
          		 Block Storage : add storage client to server protocol Sub-task Resolved Chen Liang
          3.
          		 Block Storage : add block storage server Sub-task Resolved Chen Liang
          4.
          		 Block Storage : store server state to persistent storage Sub-task Resolved Chen Liang
          5.
          		 Block Storage : add container representation to meta data Sub-task Resolved Chen Liang
          6.
          		 Block Storage : add command line tool Sub-task Resolved Chen Liang
          7.
          		 Block Storage : add cache interface Sub-task Resolved Chen Liang
          8.
          		 Block Storage : merge configuration into ozone configuration classes Sub-task Resolved Chen Liang
          9.
          		 Block Storage : add cache layer Sub-task Resolved Chen Liang
          10.
          		 Block Storage: fix TestCBlockServer test cleanup Sub-task Resolved Xiaoyu Yao
          11.
          		 Block Storage: Add Support for Direct I/O Sub-task Resolved Mukul Kumar Singh
          12.
          		 Block Storage: Add metrics for Container Flushes. Sub-task Resolved Mukul Kumar Singh
          13.
          		 Block Storage : add SCSI target access daemon Sub-task Resolved Chen Liang
          14.
          		 Block Storage : allow cblock server to be started from hdfs commandline Sub-task Resolved Chen Liang
          15.
          		 Block Storage : configurable max cache size Sub-task Open Mukul Kumar Singh
          16.
          		 Block Storage: Cblock cache should register with flusher to upload blocks to containers Sub-task Resolved Mukul Kumar Singh
          17.
          		 Block Storage:Handling flushing of incomplete block id buffers during shutdown Sub-task Resolved Mukul Kumar Singh
          18.
          		 Block Storage: Add Support for 2 BlockIDBuffers and also for periodic flush of BlockIDBuffer Sub-task Resolved Mukul Kumar Singh
          19.
          		 Block Storage: Retry Blocks should be requeued when cblock is restarted Sub-task Resolved Mukul Kumar Singh
          20.
          		 Block Storage: Fix TestCBlockCLI, failing because of " Address already in use" Sub-task Resolved Mukul Kumar Singh
          21.
          		 Block Storage: Overwrite of blocks fails Sub-task Resolved Mukul Kumar Singh
          22.
          		 Block Storage: Convert unnecessary info log levels to debug Sub-task Resolved Mukul Kumar Singh
          23.
          		 CBlocks use wrong OPT env vars Sub-task Resolved Mukul Kumar Singh
          24.
          		 Add documentation for metrics in CBlockTargetMetrics to OzoneMetrics.md Sub-task Open Mukul Kumar Singh
          25.
          		 Add markdown documentation about Cblock Sub-task Open Mukul Kumar Singh
          26.
          		 Block Storage : make the server address config more concise Sub-task Resolved Chen Liang
          27.
          		 Block Storage: Fix cblock unit test failures in TestLocalBlockCache, TestCBlockReadWrite and TestBufferManager Sub-task Open Mukul Kumar Singh
          28.
          		 Block Storage: Add flushid in place of timestamp in DirtyLog file signatures Sub-task Open Mukul Kumar Singh
          29.
          		 Block Storage: Add general abstraction for metadata store in CBlock Sub-task Open Mukul Kumar Singh
          30.
          		 Block Storage: volume creation times out while creating 3TB volume because of too many containers Sub-task Resolved Mukul Kumar Singh
          31.
          		 Block Storage: Add start-cblock.sh to quickly start cblock. Sub-task Open Mukul Kumar Singh
          32.
          		 Block Storage: Cblock should generated unique trace ID for the ops Sub-task Resolved Mukul Kumar Singh
          33.
          		 Block Storage: Change dfs.cblock.jscsi.server.address default bind address to 0.0.0.0 Sub-task Resolved Mukul Kumar Singh
          34.
          		 Block Storage: fix thread number calculation in CBlockManager Sub-task Resolved Marton Elek
          35.
          		 Block Storage: provide docker-compose file for cblock clusters Sub-task Resolved Marton Elek
          36.
          		 Block Storage: implement simple iscsi discovery in jscsi server Sub-task Resolved Marton Elek
          37.
          		 Block Storage: make the iscsi target addres configurable for discovery Sub-task Resolved Marton Elek
          38.
          		 Block Storage: Kubernetes dynamic persistent volume provisioner Sub-task Resolved Marton Elek

          Activity

            People

              aengineer Anu Engineer
              aengineer Anu Engineer
              Votes:
              0 Vote for this issue
              Watchers:
              49 Start watching this issue

              Dates

                Created:
                Updated: