[HIVE-13770] HiveServer2: Improve Thrift result set streaming when serializing thrift ResultSets in tasks by fetching the results as soon as they are available - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Add vote

Voters

Watch issue

Watchers

Convert to Issue

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: HiveServer2, JDBC
Labels:
None

Description

When serializing the Thrift result set in final task, i.e. the hive.server2.thrift.resultset.serialize.in.tasks property is set to true, HS2 does not start sending the results until the entire result set has been written to HDFS.

This is not efficient and we should find a way for HS2 to start sending the results as soon as a block of result becomes available. The advantage for this is two folds. One, the client can start consuming the results much sooner. Two, we can start reclaiming the storage space in HDFS used by a particular result set block as soon as the result set block has been successfully sent to the client.

It's worth checking if this is also the case when not serializing the Thrift result set in final task.