[BEAM-13433] beam_PostCommit_Python37 failing, potentially due to apache_beam.ml.gcp.cloud_dlp_it_test.CloudDLPIT

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Duplicate
• Affects Version/s: 2.36.0
• Fix Version/s: Not applicable
• Component/s: test-failures
• Assignee: Kyle Weaver
• Reporter: Daniel Oliveira
• Created: 10/Dec/21 04:18
• Updated: 16/Dec/21 20:54
• Resolved: 16/Dec/21 20:54

Description

It's difficult for me to test for sure, because each run seems to show slightly different errors, and sometimes the errors don't even show at all. To track this down, you need to check the gradle build scan for the test, because the raw logs are too long to find the appropriate error.

This is one that shows an error: https://ci-beam.apache.org/job/beam_PostCommit_Python37/4617/

As far as I can tell, this is the error, apparently happening due to https://github.com/apache/beam/blob/master/sdks/python/apache_beam/ml/gcp/cloud_dlp_it_test.py

Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1198, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 536, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1334, in apache_beam.runners.common._OutputProcessor.process_outputs
File "/usr/local/lib/python3.7/site-packages/apache_beam/ml/gcp/cloud_dlp.py", line 199, in process
  item={"value": element}, **self.params)
TypeError: deidentify_content() got an unexpected keyword argument 'item'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 644, in do_work
  work_executor.execute()
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 208, in execute
  op.start()
File "dataflow_worker/native_operations.py", line 38, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 39, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 44, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 54, in dataflow_worker.native_operations.NativeReadOperation.start
File "apache_beam/runners/worker/operations.py", line 348, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 215, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 707, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 708, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1200, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1281, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1198, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 536, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1334, in apache_beam.runners.common._OutputProcessor.process_outputs
File "/usr/local/lib/python3.7/site-packages/apache_beam/ml/gcp/cloud_dlp.py", line 199, in process
  item={"value": element}, **self.params)
TypeError: deidentify_content() got an unexpected keyword argument 'item' [while running 'MaskDetectedDetails/ParDo(_DeidentifyFn)']

Issue Links

• duplicates: BEAM-13388 [beam_PostCommit_Python38] [apache_beam.ml.gcp.cloud_dlp_it_test.CloudDLPIT.test_deidentification] Failure summary