Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12682

ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      On a real cluster, hdfs ec -listPolicies will always show policy state as DISABLED.

      [hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
      Erasure Coding Policies:
      ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
      ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
      ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
      ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=3, State=DISABLED]
      ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
      [hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
      XOR-2-1-1024k
      

      This is because when deserializing protobuf, the static instance of SystemErasureCodingPolicies class is first checked, and always returns the cached policy objects, which are created by default with state=DISABLED.

      All the existing unit tests pass, because that static instance that the client (e.g. ECAdmin) reads in unit test is updated by NN.

        Attachments

        1. HDFS-12682.01.patch
          58 kB
          Xiao Chen
        2. HDFS-12682.02.patch
          61 kB
          Xiao Chen
        3. HDFS-12682.03.patch
          8 kB
          Xiao Chen
        4. HDFS-12682.04.patch
          60 kB
          Xiao Chen
        5. HDFS-12682.05.patch
          61 kB
          Xiao Chen
        6. HDFS-12682.06.patch
          61 kB
          Xiao Chen
        7. HDFS-12682.07.patch
          60 kB
          Xiao Chen
        8. HDFS-12682.08.patch
          61 kB
          Xiao Chen

          Issue Links

            Activity

              People

              • Assignee:
                xiaochen Xiao Chen
                Reporter:
                xiaochen Xiao Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: