Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-4076

Schema followups

    XMLWordPrintableJSON

    Details

      Description

      This umbrella bug contains subtasks with followups for Beam schemas, which were moved from SQL to the core Java SDK and made to be type-name-based rather than coder based.

        Attachments

          Issue Links

          1.
          Review Schema API surface Sub-task Triage Needed Unassigned  
          2.
          Define & document the domain of Schema types prominently Sub-task Open Unassigned  
          3.
          Consider Schema.join to automatically produce a correct joined schema Sub-task Open Unassigned  
          4.
          Review of schema metadata vs schema types Sub-task Open Unassigned  
          5.
          FieldType should be a proper algebraic type Sub-task Open Unassigned  
          6.
          Document the SDK contract for a PCollection having a schema Sub-task Open Unassigned  
          7.
          SQL operators and primitive values should use a richer type system than SqlTypeName Sub-task Open Unassigned  
          8.
          Valildate that OutputReceiver<Row> is only allowed if the output PCollection has a schema Sub-task Open Unassigned  
          9.
          Provide automatic schema registration for BigQuery TableRows Sub-task Open Unassigned  
          10.
          Analyze FieldAccessDescriptors and drop fields that are never accessed Sub-task Open Unassigned  
          11.
          Support unknown fields in Rows Sub-task Open Unassigned  
          12.
          Schemas across pipeline modifications Sub-task Open Unassigned  
          13.
          Investigate other encoding mechanism for SchemaCoder Sub-task Open Unassigned  
          14.
          Improve performance of SchemaCoder Sub-task Resolved Reuven Lax

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m
          15.
          Enable schemas for all runners Sub-task Open Unassigned

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          16.
          isSubType isSuperType methods do not belong in Schema.FieldType Sub-task Open Unassigned  
          17.
          Create proto representation for schemas Sub-task Open Unassigned

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2.5h
          18.
          Convert should support more boxing and unboxing Sub-task Open Unassigned  
          19.
          Remove FieldType metadata Sub-task Open Unassigned  
          20.
          Beam transforms reorder fields Sub-task Open Unassigned  
          21.
          Create a better Schema builder Sub-task Open Unassigned  
          22.
          PubSubIO.writeAvros should infer beam schemas Sub-task Open Unassigned  
          23.
          KafkaIO should support inferring schemas when reading Avro Sub-task Open Unassigned

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m
          24.
          Allow selecting slices of arrays and maps Sub-task Open Unassigned  
          25.
          Add support for generics in schema inference Sub-task Open Unassigned  
          26.
          Some JDBC types do not have an equivalent Beam schema type Sub-task Open Unassigned  
          27.
          Duplication of code between JDBC Read classes Sub-task Open Unassigned  
          28.
          support schemas in state API Sub-task Open Unassigned

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          29.
          Add examples using Schema-based APIs Sub-task Open Unassigned  

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kenn Kenneth Knowles
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 195h 20m
                  195h 20m