Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11425 Cell/DBB end-to-end on the read-path
  3. HBASE-15077

Support OffheapKV write in compaction with out copying data on heap

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: regionserver, Scanners
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBASE-14832 is not enough to handle this. Doing the remaining needed here.

       if (cell instanceof ByteBufferedCell) {
      890	      out.writeShort(rowLen);
      891	      ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell) cell).getRowByteBuffer(),
      892	        ((ByteBufferedCell) cell).getRowPosition(), rowLen);
      893	      out.writeByte(fLen);
      894	      ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell) cell).getFamilyByteBuffer(),
      895	        ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
      896	      ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell) cell).getQualifierByteBuffer(),
      897	        ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
      

      We have done this but it is not really helping us!
      In ByteBufferUtils#copyBufferToStream

      public static void copyBufferToStream(OutputStream out, ByteBuffer in,
            int offset, int length) throws IOException {
          if (in.hasArray()) {
            out.write(in.array(), in.arrayOffset() + offset,
                length);
          } else {
            for (int i = 0; i < length; ++i) {
              out.write(toByte(in, offset + i));
            }
          }
        }
        

      So for DBB it is so costly op writing byte by byte reading each to on heap.
      Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset, int length), it won't help us as the underlying stream is a ByteArrayOutputStream and so we will end up in copying.

        Attachments

        1. HBASE-15077_V2.patch
          21 kB
          Anoop Sam John
        2. HBASE-15077_V2.patch
          21 kB
          Anoop Sam John
        3. HBASE-15077_V3.patch
          21 kB
          Anoop Sam John
        4. HBASE-15077.patch
          21 kB
          Anoop Sam John

          Issue Links

            Activity

              People

              • Assignee:
                anoop.hbase Anoop Sam John
                Reporter:
                anoop.hbase Anoop Sam John
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: