[HIVE-13770] HiveServer2: Improve Thrift result set streaming when serializing thrift ResultSets in tasks by fetching the results as soon as they are available

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: HiveServer2, JDBC
• Assignee: Ziyang Zhao
• Reporter: Holman Lan
• Created: 16/May/16 22:20
• Updated: 10/Oct/16 21:46

Description

When serializing the Thrift result set in final task, i.e. the hive.server2.thrift.resultset.serialize.in.tasks property is set to true, HS2 does not start sending the results until the entire result set has been written to HDFS.

This is not efficient and we should find a way for HS2 to start sending the results as soon as a block of result becomes available. The advantage for this is two folds. One, the client can start consuming the results much sooner. Two, we can start reclaiming the storage space in HDFS used by a particular result set block as soon as the result set block has been successfully sent to the client.

It's worth checking if this is also the case when not serializing the Thrift result set in final task.