Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14679

Failed to add erasure code policies with example template

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1.2
    • Fix Version/s: 3.3.0
    • Component/s: ec
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hi Hadoop developers,

       

      Trying to do some quick tests with erasure coding feature and ran into a issue on adding policies. The example on adding erasure code policies with provided template failed:

      ./bin/hdfs ec -addPolicies -policyFile /tmp/user_ec_policies.xml.template
      2019-07-30 10:35:16,447 INFO util.ECPolicyLoader: Loading EC policy file /tmp/user_ec_policies.xml.template
      Add ErasureCodingPolicy XOR-2-1-128k succeed.
      Add ErasureCodingPolicy RS-LEGACY-12-4-256k failed and error message is Codec name RS-legacy is not supported

      The issue seems due to be the mismatching codec(upper case vs lower case). The codec is in upper case in the example template[1] while all available codecs are lower case[2]. A way to fix maybe just converting the codec to lower case when parsing the policy schema. Also attached a simple patch here. 

      [1] https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/conf/user_ec_policies.xml.template#L51

      [2]https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ErasureCodeConstants.java#L28-L33

      Thanks, -yuan

        Attachments

        1. fix_adding_EC_policy_example.diff
          0.9 kB
          Yuan Zhou
        2. HDFS-14679-01.patch
          4 kB
          Yuan Zhou
        3. HDFS-14679-02.patch
          1 kB
          Yuan Zhou
        4. HDFS-14679-03.patch
          1 kB
          Yuan Zhou

          Activity

            People

            • Assignee:
              yuanzhou Yuan Zhou
              Reporter:
              yuanzhou Yuan Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: