Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11264 Common side changes for HDFS Erasure coding support
  3. HADOOP-11938

Enhance ByteBuffer version encode/decode API of raw erasure coder

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HDFS-7285
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      This enhances the ByteBuffer version of encode / decode API of raw erasure coder allowing to process variable length of inputs data.

      Description

      While investigating a test failure in TestRecoverStripedFile, it was found that raw erasure coder may be used breaking its assumptions in the ByteBuffer version encode/decode API. Originally it assumes the ByteBuffer inputs/outputs available for reading or writing always start with zero (position) and takes the whole space (limit). Below is a code sample in that assumption:

        protected static byte[][] toArrays(ByteBuffer[] buffers) {
          byte[][] bytesArr = new byte[buffers.length][];
      
          ByteBuffer buffer;
          for (int i = 0; i < buffers.length; i++) {
            buffer = buffers[i];
            if (buffer == null) {
              bytesArr[i] = null;
              continue;
            }
      
            if (buffer.hasArray()) {
              bytesArr[i] = buffer.array();
            } else {
              throw new IllegalArgumentException("Invalid ByteBuffer passed, " +
                  "expecting heap buffer");
            }
          }
      
          return bytesArr;
        }
      

        Attachments

        1. HADOOP-11938-HDFS-7285-workaround.patch
          1 kB
          Kai Zheng
        2. HADOOP-11938-HDFS-7285-v1.patch
          36 kB
          Kai Zheng
        3. HADOOP-11938-HDFS-7285-v2.patch
          44 kB
          Kai Zheng
        4. HADOOP-11938-HDFS-7285-v3.patch
          50 kB
          Kai Zheng
        5. HADOOP-11938-HDFS-7285-v4.patch
          49 kB
          Kai Zheng

          Issue Links

            Activity

              People

              • Assignee:
                drankye Kai Zheng
                Reporter:
                drankye Kai Zheng
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: