[HBASE-22309] Replace Shipper Interface with Netty's ReferenceCounted; add ExtendCell#retain/ExtendCell#release

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Assignee: Zheng Hu
• Reporter: Zheng Hu
• Created: 25/Apr/19 09:08
• Updated: 25/Jun/20 19:34

Description

We've some discussion about the Shipper interface.

/**
 * This interface denotes a scanner as one which can ship cells. Scan operation do many RPC requests
 * to server and fetch N rows/RPC. These are then shipped to client. At the end of every such batch
 * {@link #shipped()} will get called.
 */
@InterfaceAudience.Private
public interface Shipper {

  /**
   * Called after a batch of rows scanned and set to be returned to client. Any in between cleanup
   * can be done here.
   */
  void shipped() throws IOException;
}

it seems not an elegant ways...for example:
1. we want to keep the previous cell in the scanner, we must deep clone the kv before an shipping, otherwise the ship will free all the ByteBuffers, the prevCell will point to an unknown area.
2. when switch from PREAD to STREAM in a long scan, we also have accomplish this in shipped() method. if not , once we close the PREAD scanner, the un-shipped cell will also point to an unknown memory area, because the scanner closing free all ByteBuffes.
....

If we change to use refCnt to manage the RPC memory release or retain, we can just call prevCell.retain.. then it memory won't be free unless the prevCell reach the end of life and call prevCell#release. I mean we can replace all the shipper logic by using cell#release and cell#retain.

One concern is about the API, actually the ExtendCell is an pure server side type, we can make the ExtendCell extend the Netty's ReferenceCounted interface , and provide an retain() and release() methods. we won't maitain an refCnt in ExtendCell, the refCnt is still in HFileBlock. Once we encoded an ExtendCell to CellScanner, we can release the extendCell, and it will release its blocks backend... so in theory, will have no performance loss...

Anyway, that would be a big change, so maybe need to create another new feature branch to address this.

Issue Links

• relates to: HBASE-21879 Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose