[HDDS-5822] EC: Writing a large buffer to an EC file duplicates first chunk in block 1 and 2

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: EC-Branch
• Labels: pull-request-available
• Assignee: Uma Maheswara Rao G
• Reporter: Stephen O'Donnell
• Created: 05/Oct/21 20:19
• Updated: 07/Oct/21 23:27
• Resolved: 07/Oct/21 00:17

Description

If you write a large buffer of data containing several chunks of data like this:

byte[] inputData = new byte[dataLength];
    RAND.nextBytes(inputData);
    for (byte b : inputData) {
      key.write(b);
    }

Then the current EC Key logic will write the first chunk twice to block 1 and block 2 and then probably (I have not verified) drop the last chunk completely.

This is due to a bug in ECKeyOutputStream.write(…):

 int currentChunkBufferRemainingLength =
        ecChunkBufferCache.dataBuffers[blockOutputStreamEntryPool.getCurrIdx()]
            .remaining();
    int currentChunkBufferLen =
        ecChunkBufferCache.dataBuffers[blockOutputStreamEntryPool.getCurrIdx()]
            .position();
    int maxLenToCurrChunkBuffer = (int) Math.min(len, ecChunkSize);
    int currentWriterChunkLenToWrite =
        Math.min(currentChunkBufferRemainingLength, maxLenToCurrChunkBuffer);
    int pos = handleDataWrite(blockOutputStreamEntryPool.getCurrIdx(), b, off,
        currentWriterChunkLenToWrite,
        currentChunkBufferLen + currentWriterChunkLenToWrite == ecChunkSize);
    checkAndWriteParityCells(pos);

    int remLen = len - currentWriterChunkLenToWrite;
    int iters = remLen / ecChunkSize;
    int lastCellSize = remLen % ecChunkSize;
    while (iters > 0) {
      pos = handleDataWrite(blockOutputStreamEntryPool.getCurrIdx(), b, off,
          ecChunkSize, true);
      off += ecChunkSize;
      iters--;
      checkAndWriteParityCells(pos);
    }

Here we write the first chunk before entering the "iters" loop, but we forget to increment "off" which results in the same data getting written twice.

We need to add "currentWriterChunkLenToWrite" to "off" before entering the loop.

We should add a test to reproduce this issue and then add the fix.

Issue Links

• links to: GitHub Pull Request #2714