[BEAM-8365] Add project push-down capability to IO APIs

• Type: New Feature
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.18.0
• Component/s: dsl-sql
• Assignee: Kirill Kozlov
• Reporter: Kirill Kozlov
• Created: 07/Oct/19 21:10
• Updated: 13/Apr/23 10:58
• Resolved: 31/Oct/19 22:15

Description

• InMemoryTable should implement a following method:

  public PCollection<Row> buildIOReader(
      PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames);

  Which should return a `PCollection` with fields specified in `fieldNames` list.

• Create a rule to push fields used by a Calc (in projects and in a condition) down into TestTable IO.
• Updating that same Calc  (from previous step) to have a proper input and output schemes, remove unused fields.

Issue Links

• blocks: BEAM-8343 Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
• relates to: BEAM-8794 Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule
• links to: GitHub Pull Request #9743
• links to: GitHub Pull Request #9764