Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12420

TypeDescriptor information gets lost when applying multiple DoFn on Composite Transform

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: sdk-java-core
    • Labels:
      None

      Description

      While working on BEAM-12384 to preserve correctly the typeDescriptor of the Read transform. I noticed that ParDo passes the outputType information into the next transforms by taking the output of `getOutputTypeDescriptor` specified in the `DoFn`, but the subsequent transform does not get the full inputTypeDescriptor.

      There are other open questions:
      1. How can we make this easier for users who rarely override the getOutputTypeDescriptor?
      2. Where the typeDescriptor should come in the SDF case, from the Restriction?

      For more details:
      https://github.com/apache/beam/pull/14854#issuecomment-848793500

       

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                iemejia Ismaël Mejía
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: