[HAWQ-583] Extend PXF to allow plugins to support returning partial content of SELECT(column projection)

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0.0-incubating
• Component/s: PXF
• Assignee: Kavinder Dhaliwal
• Reporter: Michael Andre Pearce
• Created: 23/Mar/16 22:16
• Updated: 18/Feb/17 05:27
• Resolved: 22/Dec/16 07:14

Description

Currently PXF supports being able to push down the predicate WHERE logic to the external system to reduce the amount data needed to be retrieved.

SELECT a, b FROM external_pxf_source WHERE z < 3 AND x > 6

As such we can filter the rows returned, but currently still would have to return all the fields / complete row.

This proposal is so that we can return only the columns in SELECT part.

For data sources where it is columnar storage or selectable such as remote database that PXF can read or connect to this has advantages in the data that needs to be accessed or even transferred.

As like with the push down Filter it should be optional so that plugins that provide support can use it but others that do not, continue to work as they do.

The proposal would be for

1) create an interface created for plugins to optionally implement, where the columns needed to be returned are given to the plugin.

2) update pxf api for hawq to send columns defined in SELECT, for pxf to invoke the plugin interface and pass this information onto if provided

3) update pxf integration within hawq itself so that hawq passes this additonal information to pxf.

This Ticket is off the back of discussion on HAWQ-492.

Sub-Tasks

• Send Projection Info Data from HAWQ to PXF: Resolved
• Extend Projection info to include filter attributes: Resolved
• Update PXF Hive Resolver to return partial content: Resolved