[HDDS-6256] EC: Parity blocks are incorrectly padded with zeros to the chunk size (#3043)

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: EC-Branch
• Labels: pull-request-available
• Assignee: Uma Maheswara Rao G
• Reporter: Stephen O'Donnell
• Created: 03/Feb/22 13:02
• Updated: 07/Feb/22 11:34
• Resolved: 07/Feb/22 11:34

Description

I write a tiny key into a 3-2 bucket. Container group is still open - this looks good 3 containers as expected:

 Replicas: [State: OPEN; ReplicaIndex: 4; Origin: 3fc2c737-83b1-4485-91ac-b771e41fa1dd; Location: 3fc2c737-83b1-4485-91ac-b771e41fa1dd/ozone_datanode_1.ozone_default,
State: OPEN; ReplicaIndex: 5; Origin: 1be82529-d5f2-4c6b-8573-0bb75863ded1; Location: 1be82529-d5f2-4c6b-8573-0bb75863ded1/ozone_datanode_4.ozone_default,
State: OPEN; ReplicaIndex: 1; Origin: e1d6ca08-d5eb-4e8a-b42a-bf1257e9f673; Location: e1d6ca08-d5eb-4e8a-b42a-bf1257e9f673/ozone_datanode_2.ozone_default]

In the JSON:

 

"bytesUsed" : 177,
"replicaIndex" : 1
...
"bytesUsed" : 1048576,
"replicaIndex" : 5
...
"bytesUsed" : 1048576,
"replicaIndex" : 4 

So the parity should only have 177 bytes too, but it appears to have 1MB. Checking the block file on disk, it is indeed 1MB:

 bash-4.2$ cd /data/hdds/hdds/CID-137befd9-f6ec-430b-b0e0-ee683ac1d987/current/containerDir0/3/
bash-4.2$ ls
chunks metadata
bash-4.2$ cd chunks/
bash-4.2$ ls
109611004723200004.block
bash-4.2$ ls -al
total 1032
drwxr-xr-x 2 hadoop users 4096 Feb 2 22:08 .
drwxr-xr-x 4 hadoop users 4096 Feb 2 22:08 ..
rw-rr- 1 hadoop users 1048576 Feb 2 22:08 109611004723200004.block

So something is not correct with the write path here. Checking the block contents, its all padded with zeros:

 bash-4.2$ hexdump -C 109611004723200004.block 
00000000 e4 e5 e6 1a 10 1a 10 1a e4 07 24 25 21 d4 24 d3 |..........$%!.$.|
00000010 25 da 2c 06 16 16 e4 07 24 25 21 d4 24 d3 25 da |%.,.....$%!.$.%.|
00000020 2c eb 27 db 12 1b 24 25 21 d4 24 d3 25 da 2c eb |,.'...$%!.$.%.,.|
00000030 27 db 12 1b 24 25 25 db d5 d4 21 d2 06 22 23 10 |'...$%%...!.."#.|
00000040 10 16 16 10 07 27 db 12 1b 24 25 21 d4 24 d1 23 |.....'...$%!.$.#|
00000050 2c 06 22 22 10 10 16 16 10 07 27 db 12 1b d0 21 |,.""......'....!|
00000060 d4 da 2c db 2e 23 22 27 28 06 22 22 10 e5 16 16 |..,..#"'(.""....|
00000070 e4 07 27 db 12 1b d4 24 24 d1 25 d7 23 da 06 22 |..'....$$.%.#.."|
00000080 22 10 e5 16 16 e5 07 27 db 12 1b d4 24 24 2e 25 |"......'....$$.%|
00000090 d8 2c 23 2e da 06 e4 17 e5 1a e4 12 e3 1a e4 12 |.,#.............|
000000a0 10 1a e5 07 e6 e4 e7 11 d5 13 13 10 10 e3 d7 e3 |................|
000000b0 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00100000

Issue Links

• links to: GitHub Pull Request #3043