[BEAM-14100] Dataflow interactive runner throws exception when there is BigQuery I/O in the pipeline

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Duplicate
• Affects Version/s: 2.35.0, 2.36.0, 2.37.0
• Fix Version/s: Not applicable
• Component/s: runner-py-interactive
• Assignee: Ning
• Reporter: Ning
• Created: 14/Mar/22 20:34
• Updated: 16/Mar/22 21:26
• Resolved: 16/Mar/22 21:08

Description

~/apache-beam-2.37.0/lib/python3.7/site-packages/apache_beam/coders/coders.py in <lambda>(x)
    800     protocol = pickle.HIGHEST_PROTOCOL
    801     return coder_impl.CallbackCoderImpl(
--> 802         lambda x: dumps(x, protocol), pickle.loads)
    803 
    804   def as_deterministic_coder(self, step_label, error_message=None):

TypeError: can't pickle generator objects [while running '[1]: ReadFromBigQuery/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction'] 

Issue occurred since 2.35.0.

After debugging, the generator was introduced in https://github.com/apache/beam/pull/15610.

Issue Links

• is duplicated by: BEAM-14112 ReadFromBigQuery cannot be used with the interactive runner
• links to: GitHub Pull Request #17093