[BEAM-8418] Fix handling of Impulse transform in Dataflow runner.

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.17.0
• Component/s: sdk-py-core
• Assignee: Valentyn Tymofieiev
• Reporter: Valentyn Tymofieiev
• Created: 17/Oct/19 02:00
• Updated: 24/Jul/20 19:57
• Resolved: 22/Oct/19 23:55

Description

Following pipeline fails on Dataflow runner unless we use beam_fn_api experiment.

class NoOpDoFn(beam.DoFn):
  def process(self, element):
    return element

p = beam.Pipeline(options=pipeline_options)
_ = p | beam.Impulse() | beam.ParDo(NoOpDoFn())
result = p.run()

The reason is that we encode Impluse payload using url-escaping in [1], while Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF runner expects URL escaping.

We should fix or reconcile the encoding in non-FnAPI path, and add a ValidatesRunner test that catches this error.

[1] https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633

Issue Links

• links to: GitHub Pull Request #9822