[HDFS-12682] ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: erasure-coding
• Labels: hdfs-ec-3.0-must-do
• Target Version/s: 3.0.0
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Xiao Chen
• Reporter: Xiao Chen
• Created: 18/Oct/17 22:24
• Updated: 03/Nov/17 04:48
• Resolved: 03/Nov/17 04:30

Description

On a real cluster, hdfs ec -listPolicies will always show policy state as DISABLED.

[hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
Erasure Coding Policies:
ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=3, State=DISABLED]
ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
[hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
XOR-2-1-1024k

This is because when deserializing protobuf, the static instance of SystemErasureCodingPolicies class is first checked, and always returns the cached policy objects, which are created by default with state=DISABLED.

All the existing unit tests pass, because that static instance that the client (e.g. ECAdmin) reads in unit test is updated by NN.

Issue Links

• contains: HDFS-12686 Erasure coding system policy state is not correctly saved and loaded during real cluster restart
• relates to: HDFS-11565 Use compact identifiers for built-in ECPolicies in HdfsFileStatus
• relates to: HDFS-12258 ec -listPolicies should list all policies in system, no matter it's enabled or disabled