Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9807

Add an optional StorageID to writes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha2
    • 3.0.0-alpha4
    • None
    • None
    • Reviewed

    Description

      The BlockPlacementPolicy considers specific storages, but when the replica is written the DN VolumeChoosingPolicy is unaware of any preference or constraints from other policies affecting placement. This limits heterogeneity to the declared storage types, which are treated as fungible within the target DN. It should be possible to influence or constrain the DN policy to select a particular storage.

      Attachments

        1. HDFS-9807.001.patch
          139 kB
          Ewan Higgs
        2. HDFS-9807.002.patch
          131 kB
          Christopher Douglas
        3. HDFS-9807.003.patch
          147 kB
          Ewan Higgs
        4. HDFS-9807.004.patch
          147 kB
          Ewan Higgs
        5. HDFS-9807.005.patch
          148 kB
          Ewan Higgs
        6. HDFS-9807.006.patch
          128 kB
          Ewan Higgs
        7. HDFS-9807.007.patch
          143 kB
          Ewan Higgs
        8. HDFS-9807.008.patch
          143 kB
          Virajith Jalaparti
        9. HDFS-9807.009.patch
          143 kB
          Virajith Jalaparti
        10. HDFS-9807.010.patch
          141 kB
          Virajith Jalaparti

        Issue Links

          breaks

          Sub-task - The sub-task of the issue HDFS-12151 Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-11956 Do not require a storage ID or target storage IDs when writing a block

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          causes

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-13941 make storageId in BlockPoolTokenSecretManager.checkAccess optional

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HDFS-11464 Improve the selection in choosing storage for blocks

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-12207 A few DataXceiver#writeBlock cleanups related to optional storage IDs and types

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-14330 Consider StorageID to choose volume

          • Major - Major loss of function.
          • Open
          relates to

          Sub-task - The sub-task of the issue HDFS-6708 StorageType should be encoded in the block token

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-9806 Allow HDFS block replicas to be provided by an external storage system

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-11419 BlockPlacementPolicyDefault is choosing datanode in an inefficient way

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              ehiggs Ewan Higgs
              cdouglas Christopher Douglas
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: