Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12442

RenameFields disregards renames in nested fields with DataflowRunner

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: 2.26.0
    • Fix Version/s: 2.32.0
    • Component/s: sdk-java-core
    • Labels:
      None

      Description

      Summary - When applying a `RenameFields` followed by BigQueryUtils.toTableRow(Row), renames on nested fields were not being applied when using DataflowRunner. DirectRunner appeared to be fine. This led to a failed BigQueryIO.Write.

      Expected Behaviour - (field0_1.field1_0, nestedStringField) -> field0_1.nestedStringField and successful BQ insert
      Actual Behaviour - (field0_1.field1_0, nestedStringField) -> field0_1.field1_0 and failed BQ insert because field0_1.field1_0 was not in the target BQ table

      Thread from Users mailing list - https://lists.apache.org/thread.html/rd33a6f2bd8454cf650d70795bcdbbb873f988a557133d3bdbfc56cfc%40%3Cuser.beam.apache.org%3E

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mouyang Matthew Ouyang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h