Here is the paper of SimpleRegenerating Code.
http://arxiv.org/pdf/1109.0264

I think one thing we need to do is to refactor the raid.Encoder and raid.Decoder. So they are generic to any ErasureCode. That way we can easily add different codes to Raid.

It will also be nice to have this code be backwards compatible with existing Reed-Solomon parity files. If there is an existing Reed-Solomon parity file, the code can identify that by counting the number of parity blocks with the expected number of Reed-Solomon parity files. This is doable because the additional XOR parity blocks will increase the total number of parity blocks by a deterministic number. Thus this code will be able to handle existing Reed-Solomon parity files and will generate new files with additional XOR blocks.

I can see that backwards compatibility would be crucial for a deployed system. It is not always clear how to find if a parity block is a simple parity or an RS parity just by counting since the config files might have different number of simple parities (our default kept the total number of parities to 4 by having two RS and two 6 degree XORs) to keep the same storage overhead as a (14,10) Reed Solomon.

I think a cleaner way to understand what each parity is, can be done through the meta data file or the folder it is in (right now how do you distinguish simple XOR to RS parities)?

After some discussion, it seems that the easiest method to ensure backwards compatibility is to recognize if this is a Reed-Solomon coded packet or an SRC coded packet by the size of the parity file. If we use stripe size 10, 4 RS packets and 2 simple XORs then the parity file should have size 64MB*6 while for RS coded files it should be 64MB*4. We will implement this distinction in Processfile and run different decoder functions accordingly.

There is another way of doing it:
I will add a new erasure code type called SRC to ErasureCodeType (which has XOR, RS now) and start storing SRC coded files in /raidsrc (RS files stored in /raidrs, XOR in /raid). When a file corruption is detected
and recoverBlockToFile is called, the first thing to do is to check whether the file is a parity file or a source file. By looking at the location it can be easily determined whether this is a parity file and if so which type. Now if its not a parity file, then it is a source file and we need to determine its corresponding parity file. This can be done by checking for a parity file first in /raidsrc, and then in /raidrs and /raid to find out where it is located. That way we can find the parity file too.
The same thing can be done by determining the filesize, for which we still need to search for the parity file by going to /raidrs or /raid; so I think the above approach is a little bit cleaner.
For reconstructing the file, in either approach, we need to pass the ErasureCodeType all the way till the decoder and encoder.

Assign to Weiyan because he is working on this now.

Any updates on this issue? We're interested in trying this out to save space on our cold files.

Chris, Sorry for the late reply, we are actually not working on this because of the bandwidth. For cold files, we still use (14, 10) Reed Solomon to save spaces right now.

Okay, we're very interested in SRC in our clusters and we will work on this feature if we can get some momentum. Either way I'll keep you posted.

Chris, our VLDB paper has an evolution of Simple regenerating codes that we called Locally Repairable Codes, (LRC) see our paper
http://smahesh.com/HadoopUSC/

You can contact me for more details if you are interested.

Hey，i m a new learner about hadoop and hdfs-raid，i got a question ，why doesnt the raid codes be merged into the trunk since release 2.0.2？

HDFS-RAID has been removed and native EC support is available at HDFS-7285. Closing as won't fix.

The other note is that we have a JIRA to add LRC support (HDFS-7345) but it's patent encumbered by Microsoft.