[BEAM-4076] Schema followups

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: beam-model, sdk-java-core
• Labels: Clarified
• Assignee: Unassigned
• Reporter: Kenneth Knowles
• Created: 14/Apr/18 23:53
• Updated: 03/Jun/22 21:12

Description

This umbrella bug contains subtasks with followups for Beam schemas, which were moved from SQL to the core Java SDK and made to be type-name-based rather than coder based.

Issue Links

• incorporates: BEAM-9859 Support joda time and java time Duration in schema inference
• is a parent of: BEAM-10878 Dataflow fails to deserialize logical type with static schema
• is a parent of: BEAM-10765 Support non-homogenous nested types in SchemaProvider implementations
• is related to: BEAM-10388 Add support for self-referential Beam schemas
• links to: GitHub Pull Request #5498
• links to: GitHub Pull Request #5545
• links to: GitHub Pull Request #5941
• links to: GitHub Pull Request #5953
• links to: GitHub Pull Request #5955
• links to: GitHub Pull Request #6072
• links to: GitHub Pull Request #7500
• links to: GitHub Pull Request #7635
• links to: GitHub Pull Request #7928
• links to: GitHub Pull Request #11041

Sub-Tasks

• Define & document the domain of Schema types prominently: Open
• Consider Schema.join to automatically produce a correct joined schema: Open
• Review of schema metadata vs schema types: Triage Needed
• FieldType should be a proper algebraic type: Open
• Document the SDK contract for a PCollection having a schema: Open
• SQL operators and primitive values should use a richer type system than SqlTypeName: Open
• Valildate that OutputReceiver<Row> is only allowed if the output PCollection has a schema: Open
• Provide automatic schema registration for BigQuery TableRows: Open
• Analyze FieldAccessDescriptors and drop fields that are never accessed: Open
• Support unknown fields in Rows: Open
• Schemas across pipeline modifications: Open
• Investigate other encoding mechanism for SchemaCoder: Open
• isSubType isSuperType methods do not belong in Schema.FieldType: Open
• Convert should support more boxing and unboxing: Open
• Remove FieldType metadata: Open
• Beam transforms reorder fields: Open
• Create a better Schema builder: Open
• PubSubIO.writeAvros should infer beam schemas: Open
• Allow selecting slices of arrays and maps: Open
• Add support for generics in schema inference: Open
• Some JDBC types do not have an equivalent Beam schema type: Open
• Duplication of code between JDBC Read classes: Open
• support schemas in state API: Open
• Add examples using Schema-based APIs: Open