[BEAM-4076] Schema followups

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: beam-model, sdk-java-core
• Labels: Clarified
• Assignee: Unassigned
• Reporter: Kenneth Knowles
• Created: 14/Apr/18 23:53
• Updated: 03/Jun/22 21:12

Description

This umbrella bug contains subtasks with followups for Beam schemas, which were moved from SQL to the core Java SDK and made to be type-name-based rather than coder based.

Issue Links

• incorporates: BEAM-9859 Support joda time and java time Duration in schema inference
• is a parent of: BEAM-10878 Dataflow fails to deserialize logical type with static schema
• is a parent of: BEAM-10765 Support non-homogenous nested types in SchemaProvider implementations
• is related to: BEAM-10388 Add support for self-referential Beam schemas
• links to: GitHub Pull Request #5498
• links to: GitHub Pull Request #5545
• links to: GitHub Pull Request #5941
• links to: GitHub Pull Request #5953
• links to: GitHub Pull Request #5955
• links to: GitHub Pull Request #6072
• links to: GitHub Pull Request #7500
• links to: GitHub Pull Request #7635
• links to: GitHub Pull Request #7928
• links to: GitHub Pull Request #11041

Sub-Tasks

• Refactor builder field nullability: Resolved
• Review Schema API surface: Resolved
• Define & document the domain of Schema types prominently: Open
• Consider Schema.join to automatically produce a correct joined schema: Open
• Review of schema metadata vs schema types: Triage Needed
• remove RowSqlTypeBuilder: Resolved
• FieldType should be a proper algebraic type: Open
• Find remaining uses of rowType and RowType, etc, and make them Schema as appropriate: Resolved
• Document the SDK contract for a PCollection having a schema: Open
• SQL operators and primitive values should use a richer type system than SqlTypeName: Open
• Valildate that OutputReceiver<Row> is only allowed if the output PCollection has a schema: Open
• SchemaRegistry should support a ServiceLoader interface: Resolved
• Create a lazy row on top of a generic Getter interface: Resolved
• Provide automatic schema registration for POJOs: Resolved
• Provide automatic schema registration for AVROs: Resolved
• Provide automatic schema registration for Protos: Resolved
• Provide automatic schema registration for BigQuery TableRows: Open
• Analyze FieldAccessDescriptors and drop fields that are never accessed: Open
• Support unknown fields in Rows: Open
• Schemas across pipeline modifications: Open
• Investigate other encoding mechanism for SchemaCoder: Open
• Create a library of useful transforms that use schemas: Resolved
• Improve performance of SchemaCoder: Resolved
• Enable schemas for all runners: Resolved
• Move Nexmark and SQL to use the new Schema framework: Resolved
• Schemas do not work on Dataflow runner of FnApi Runner: Resolved
• Allow users to annotate POJOs and JavaBeans for richer functionality: Resolved
• Create automatic schema registration for AutoValue classes.: Resolved
• Generated row object for POJOs, Avros, and JavaBeans should work if the wrapped class is package private: Resolved
• Allow textual selection syntax for schema fields: Resolved
• Nested collection types cause NullPointerException when converting to a POJO: Resolved
• ParDo should allow any type with a compatible registered schema in the @Element parameter: Resolved
• Support schemas in BigQueryIO.Write: Resolved
• BigQueryIO.Read should automatically produce schemas: Triage Needed
• The JdbcIO source should produce schemas: Triage Needed
• The JdbcIO sink should accept schemas: Triage Needed
• isSubType isSuperType methods do not belong in Schema.FieldType: Open
• Create LogicalType for Schema fields: Resolved
• Create proto representation for schemas: Resolved
• Support lazy iterables in schemas: Triage Needed
• Select transform has non-intuitive semantics: Resolved
• Remove KV from Schema transforms: Resolved
• Convert should support more boxing and unboxing: Open
• Add transforms for modifying schemas: Resolved
• Remove FieldType metadata: Open
• Beam transforms reorder fields: Open
• Select need the ability to rename fields: Resolved
• Create a better Schema builder: Open
• PubSubIO.writeAvros should infer beam schemas: Open
• KafkaIO should support inferring schemas when reading Avro: Resolved
• Allow selecting slices of arrays and maps: Open
• Add support for generics in schema inference: Open
• Select should support nested schema selectors: Resolved
• Some JDBC types do not have an equivalent Beam schema type: Open
• Duplication of code between JDBC Read classes: Open
• Support Avro dates in Schemas: Resolved
• Support Avro enums in Schemas: Resolved
• Protobuf Beam Schema support: Resolved
• The Row object needs better builders: Resolved
• ByteBuddy Schema code does not properly handle null values: Resolved
• support schemas in state API: Open
• Schema Select does not properly handle nested nullable fields: Resolved
• Coder inference should be disabled for Row types: Triage Needed
• Add examples using Schema-based APIs: Open
• Support Dataflow update when schemas are used: Triage Needed