[BEAM-7917] Python datastore v1new fails on retry

• Type: Bug
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.14.0
• Fix Version/s: 2.18.0
• Component/s: io-py-gcp, runner-dataflow
• Labels: pull-request-available
• Environment: Python 3.7 on Dataflow
• Assignee: Dmytro Sadovnychyi
• Reporter: Dmytro Sadovnychyi
• Created: 07/Aug/19 13:41
• Updated: 13/Apr/23 10:58
• Resolved: 01/Nov/19 21:47

Description

Traceback (most recent call last):
File "apache_beam/runners/common.py", line 782, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 454, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/datastoreio.py", line 334, in process
self._flush_batch()
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/datastoreio.py", line 349, in _flush_batch
throttle_delay=util.WRITE_BATCH_TARGET_LATENCY_MS // 1000)
File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/retry.py", line 197, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/helper.py", line 99, in write_mutations
batch.commit()
File "/usr/local/lib/python3.7/site-packages/google/cloud/datastore/batch.py", line 271, in commit
raise ValueError("Batch must be in progress to commit()")
ValueError: Batch must be in progress to commit()

Issue Links

• links to: GitHub Pull Request #9294