Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12090

Handling writes from HDFS to Provided storages

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      HDFS-9806 introduces the concept of PROVIDED storage, which makes data in external storage systems accessible through HDFS. However, HDFS-9806 is limited to data being read through HDFS. This JIRA will deal with how data can be written to such PROVIDED storages from HDFS.

        Attachments

        1. External-SyncService-CreateFile.001.png
          105 kB
          Ewan Higgs
        2. HDFS-12090.0001.patch
          411 kB
          Ewan Higgs
        3. HDFS-12090.0000.patch
          822 kB
          Ewan Higgs
        4. HDFS-12090-Functional-Specification.003.pdf
          101 kB
          Ewan Higgs
        5. HDFS-12090-Functional-Specification.002.pdf
          89 kB
          Ewan Higgs
        6. HDFS-12090-Functional-Specification.001.pdf
          98 kB
          Ewan Higgs
        7. HDFS-12090-design.001.pdf
          737 kB
          Virajith Jalaparti

          Issue Links

          1.
          [PROVIDED Phase 2] Encode the BlockAlias in the client protocol Sub-task Closed Ewan Higgs
          2.
          [PROVIDED Phase 2] Refactor FsDatasetImpl to use the BlockAlias from ClientProtocol for PROVIDED blocks. Sub-task Open Ewan Higgs
          3.
          [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup mounts Sub-task Resolved Ewan Higgs
          4.
          [DISCUSS] Provided Storage BlockAlias Refactoring Sub-task Open Unassigned
          5.
          [PROVIDED Phase 2] Provided Storage Mount Manager (PSMM) mount Sub-task Patch Available Ewan Higgs
          6.
          [PROVIDED Phase 2] Block Bit Field Allocation of Provided Storage Sub-task Resolved Unassigned
          7.
          [PROVIDED Phase 2] Add a pluggable policy for selecting locations for Provided files. Sub-task Open Unassigned
          8.
          [PROVIDED Phase 2] The DatanodeProtocol should be have DNA_BACKUP to backup blocks Sub-task Resolved Ewan Higgs
          9.
          [PROVIDED Phase 2] Implement DNA_BACKUP command in Datanode Sub-task Resolved Ewan Higgs
          10.
          [PROVIDED Phase 2] Implement pause/resume for syncservice Sub-task Open Unassigned
          11.
          [PROVIDED Phase 2] Multipart Uploader API Sub-task Resolved Ewan Higgs
          12.
          [PROVIDED Phase 2] Implement full resync command for syncservice Sub-task Open Unassigned
          13.
          [PROVIDED Phase 2] Provided ReplicaMap should be LRU with separate lookup from normal Replicas Sub-task Open Virajith Jalaparti
          14.
          Add specification of Multipart Upload API to FS specification, with contract tests Sub-task Resolved Ewan Higgs
          15.
          HDFS Classes used for implementation of Multipart uploads to move to hadoop-common JAR Sub-task Resolved Ewan Higgs
          16.
          [PROVIDED Phase 2] Scheduler in the NN for distributing DNA_BACKUP work. Sub-task Resolved Ewan Higgs
          17.
          [PROVIDED Phase 2] Teach BlockAliasMap.Writer `remove` method. Sub-task Resolved Ewan Higgs
          18.
          Multipart uploaders to be created through API call to FileSystem/FileContext, not service loader Sub-task Open Steve Loughran
          19.
          multipart upload to HDFS to support 0 byte upload Sub-task Resolved Ewan Higgs
          20.
          Multipart Uploader APIs to be marked as private/unstable in 3.2.0 Sub-task Resolved Steve Loughran
          21.
          Implement Multipart-aware distcp Sub-task Open Unassigned

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                virajith Virajith Jalaparti
              • Votes:
                2 Vote for this issue
                Watchers:
                31 Start watching this issue

                Dates

                • Created:
                  Updated: