Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2937 Fn API combiner support w/ lifting to PGBK
  3. BEAM-10266

Pipeline proto seems to be incorrect for Combine.GroupedValues

    XMLWordPrintableJSON

    Details

      Description

      It looks like CombineTest$BasicTests#testHotKeyCombining on Dataflow (and possibly other runners) is creating an invalid pipeline proto since the transform doesn't an environment (and possible a spec):

      I0610 16:05:23.791430   14054 fnapi_instruction_graph_rewriter.cc:230]   transforms {
      I0610 16:05:23.791402   14054 fnapi_instruction_graph_rewriter.cc:230]     key: "HotMean/PostCombine/Combine.GroupedValues"
      I0610 16:05:23.791404   14054 fnapi_instruction_graph_rewriter.cc:230]     value {
      I0610 16:05:23.791406   14054 fnapi_instruction_graph_rewriter.cc:230]       inputs {
      I0610 16:05:23.791408   14054 fnapi_instruction_graph_rewriter.cc:230]         key: "org.apache.beam.sdk.values.PCollection.<init>:400#56b99bb29b40d50c"
      I0610 16:05:23.791410   14054 fnapi_instruction_graph_rewriter.cc:230]         value: "HotMean/PostCombine/GroupByKey.out"
      I0610 16:05:23.791412   14054 fnapi_instruction_graph_rewriter.cc:230]       }
      I0610 16:05:23.791414   14054 fnapi_instruction_graph_rewriter.cc:230]       outputs {
      I0610 16:05:23.791416   14054 fnapi_instruction_graph_rewriter.cc:230]         key: "org.apache.beam.sdk.values.PCollection.<init>:400#4fa4d31096ca160c"
      I0610 16:05:23.791419   14054 fnapi_instruction_graph_rewriter.cc:230]         value: "HotMean/PostCombine/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous).output"
      I0610 16:05:23.791421   14054 fnapi_instruction_graph_rewriter.cc:230]       }
      I0610 16:05:23.791423   14054 fnapi_instruction_graph_rewriter.cc:230]       unique_name: "HotMean/PostCombine/Combine.GroupedValues"
      I0610 16:05:23.791426   14054 fnapi_instruction_graph_rewriter.cc:230]     }
      I0610 16:05:23.791428   14054 fnapi_instruction_graph_rewriter.cc:230]   }

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lcwik Luke Cwik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: