Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12090

Handling writes from HDFS to Provided storages

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None

      Description

      HDFS-9806 introduces the concept of PROVIDED storage, which makes data in external storage systems accessible through HDFS. However, HDFS-9806 is limited to data being read through HDFS. This JIRA will deal with how data can be written to such PROVIDED storages from HDFS.

        Attachments

        1. External-SyncService-CreateFile.001.png
          105 kB
          Ewan Higgs
        2. HDFS-12090.0001.patch
          411 kB
          Ewan Higgs
        3. HDFS-12090.0000.patch
          822 kB
          Ewan Higgs
        4. HDFS-12090-Functional-Specification.003.pdf
          101 kB
          Ewan Higgs
        5. HDFS-12090-Functional-Specification.002.pdf
          89 kB
          Ewan Higgs
        6. HDFS-12090-Functional-Specification.001.pdf
          98 kB
          Ewan Higgs
        7. HDFS-12090-design.001.pdf
          737 kB
          Virajith Jalaparti

        Issue Links

        1.
        [PROVIDED Phase 2] Encode the BlockAlias in the client protocol Sub-task Closed Ewan Higgs Actions
        2.
        [PROVIDED Phase 2] Refactor FsDatasetImpl to use the BlockAlias from ClientProtocol for PROVIDED blocks. Sub-task Open Ewan Higgs Actions
        3.
        [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup mounts Sub-task Resolved Ewan Higgs Actions
        4.
        [DISCUSS] Provided Storage BlockAlias Refactoring Sub-task Open Unassigned Actions
        5.
        [PROVIDED Phase 2] Provided Storage Mount Manager (PSMM) mount Sub-task Patch Available Ewan Higgs Actions
        6.
        [PROVIDED Phase 2] Block Bit Field Allocation of Provided Storage Sub-task Resolved Unassigned Actions
        7.
        [PROVIDED Phase 2] Add a pluggable policy for selecting locations for Provided files. Sub-task Open Unassigned Actions
        8.
        [PROVIDED Phase 2] The DatanodeProtocol should be have DNA_BACKUP to backup blocks Sub-task Resolved Ewan Higgs Actions
        9.
        [PROVIDED Phase 2] Implement DNA_BACKUP command in Datanode Sub-task Resolved Ewan Higgs Actions
        10.
        [PROVIDED Phase 2] Implement pause/resume for syncservice Sub-task Open Unassigned Actions
        11.
        [PROVIDED Phase 2] Multipart Uploader API Sub-task Resolved Ewan Higgs Actions
        12.
        [PROVIDED Phase 2] Implement full resync command for syncservice Sub-task Open Unassigned Actions
        13.
        [PROVIDED Phase 2] Provided ReplicaMap should be LRU with separate lookup from normal Replicas Sub-task Open Virajith Jalaparti Actions
        14.
        Add specification of Multipart Upload API to FS specification, with contract tests Sub-task Resolved Ewan Higgs Actions
        15.
        HDFS Classes used for implementation of Multipart uploads to move to hadoop-common JAR Sub-task Resolved Ewan Higgs Actions
        16.
        [PROVIDED Phase 2] Scheduler in the NN for distributing DNA_BACKUP work. Sub-task Resolved Ewan Higgs Actions
        17.
        [PROVIDED Phase 2] Teach BlockAliasMap.Writer `remove` method. Sub-task Resolved Ewan Higgs Actions
        18.
        Multipart uploaders to be created through API call to FileSystem/FileContext, not service loader Sub-task Reopened Steve Loughran Actions
        19.
        multipart upload to HDFS to support 0 byte upload Sub-task Resolved Ewan Higgs Actions
        20.
        Multipart Uploader APIs to be marked as private/unstable in 3.2.0 Sub-task Resolved Steve Loughran Actions
        21.
        Implement Multipart-aware distcp Sub-task Open Unassigned Actions
        22.
        remove META-INF/services/org.apache.hadoop.fs.MultipartUploader file Sub-task Open Unassigned Actions

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              virajith Virajith Jalaparti

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m

                  Issue deployment