Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12682

ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED

    XMLWordPrintableJSON

Details

    • Incompatible change, Reviewed

    Description

      On a real cluster, hdfs ec -listPolicies will always show policy state as DISABLED.

      [hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
      Erasure Coding Policies:
      ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
      ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
      ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
      ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=3, State=DISABLED]
      ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
      [hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
      XOR-2-1-1024k
      

      This is because when deserializing protobuf, the static instance of SystemErasureCodingPolicies class is first checked, and always returns the cached policy objects, which are created by default with state=DISABLED.

      All the existing unit tests pass, because that static instance that the client (e.g. ECAdmin) reads in unit test is updated by NN.

      Attachments

        1. HDFS-12682.01.patch
          58 kB
          Xiao Chen
        2. HDFS-12682.02.patch
          61 kB
          Xiao Chen
        3. HDFS-12682.03.patch
          8 kB
          Xiao Chen
        4. HDFS-12682.04.patch
          60 kB
          Xiao Chen
        5. HDFS-12682.05.patch
          61 kB
          Xiao Chen
        6. HDFS-12682.06.patch
          61 kB
          Xiao Chen
        7. HDFS-12682.07.patch
          60 kB
          Xiao Chen
        8. HDFS-12682.08.patch
          61 kB
          Xiao Chen

        Issue Links

          Activity

            People

              xiaochen Xiao Chen
              xiaochen Xiao Chen
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: