Details
-
Improvement
-
Status: In Progress
-
P3
-
Resolution: Unresolved
-
2.13.0
-
None
-
None
Description
GameStats example is failing with following logs
***************************************************** * Running GameStats with DirectRunner ***************************************************** INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner. /usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1137: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported method_to_use = self._compute_method(p, p.options) INFO:root:Running pipeline with DirectRunner. INFO:oauth2client.transport:Attempting refresh to obtain initial access_token INFO:oauth2client.client:Refreshing access_token INFO:root:Created table google.com:clouddfe.goenka_python_validations.game_stats_teams with schema <TableSchema fields: [<TableFieldSchema fields: [] mode: u'NULLABLE' name: u'processing_time' type: u'STRING'>, <TableFieldSchema fields: [] mode: u'NULLABLE' name: u'total_score' type: u'INTEGER'>, <TableFieldSchema fields: [] mode: u'NULLABLE' name: u'window_start' type: u'STRING'>, <TableFieldSchema fields: [] mode: u'NULLABLE' name: u'team' type: u'STRING'>]>. Result: <Table creationTime: 1559341252301 etag: u'tEQeN0Sr4FvttBRZuXn7zw==' id: u'google.com:clouddfe:goenka_python_validations.game_stats_teams' kind: u'bigquery#table' lastModifiedTime: 1559341252336 location: u'US' numBytes: 0 numLongTermBytes: 0 numRows: 0 schema: <TableSchema fields: [<TableFieldSchema fields: [] mode: u'NULLABLE' name: u'processing_time' type: u'STRING'>, <TableFieldSchema fields: [] mode: u'NULLABLE' name: u'total_score' type: u'INTEGER'>, <TableFieldSchema fields: [] mode: u'NULLABLE' name: u'window_start' type: u'STRING'>, <TableFieldSchema fields: [] mode: u'NULLABLE' name: u'team' type: u'STRING'>]> selfLink: u'https://www.googleapis.com/bigquery/v2/projects/google.com:clouddfe/datasets/goenka_python_validations/tables/game_stats_teams' tableReference: <TableReference datasetId: u'goenka_python_validations' projectId: u'google.com:clouddfe' tableId: u'game_stats_teams'> type: u'TABLE'>. INFO:oauth2client.transport:Attempting refresh to obtain initial access_token INFO:oauth2client.client:Refreshing access_token ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception. Traceback (most recent call last): File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call finish_state) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call evaluator.process_element(value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element self.runner.process(element) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process self._reraise_augmented(exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented raise_with_traceback(new_exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process return self.do_fn_invoker.invoke_process(windowed_value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process windowed_value, additional_args, additional_kwargs, output_processor) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window side_inputs = [si[window] for si in self.side_inputs] File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__ _FilteringIterable(self._iterable, target_window), self._view_options) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable return options['data'].view_fn(it) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda> lambda iterable: from_runtime_iterable(iterable, view_options)) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable len(head), str(head[0]), str(head[1]))) ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter'] ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception. Traceback (most recent call last): File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call finish_state) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call evaluator.process_element(value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element self.runner.process(element) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process self._reraise_augmented(exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented raise_with_traceback(new_exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process return self.do_fn_invoker.invoke_process(windowed_value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process windowed_value, additional_args, additional_kwargs, output_processor) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window side_inputs = [si[window] for si in self.side_inputs] File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__ _FilteringIterable(self._iterable, target_window), self._view_options) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable return options['data'].view_fn(it) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda> lambda iterable: from_runtime_iterable(iterable, view_options)) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable len(head), str(head[0]), str(head[1]))) ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter'] ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception. Traceback (most recent call last): File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call finish_state) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call evaluator.process_element(value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element self.runner.process(element) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process self._reraise_augmented(exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented raise_with_traceback(new_exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process return self.do_fn_invoker.invoke_process(windowed_value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process windowed_value, additional_args, additional_kwargs, output_processor) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window side_inputs = [si[window] for si in self.side_inputs] File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__ _FilteringIterable(self._iterable, target_window), self._view_options) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable return options['data'].view_fn(it) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda> lambda iterable: from_runtime_iterable(iterable, view_options)) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable len(head), str(head[0]), str(head[1]))) ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter'] ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception. Traceback (most recent call last): File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call finish_state) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call evaluator.process_element(value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element self.runner.process(element) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process self._reraise_augmented(exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented raise_with_traceback(new_exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process return self.do_fn_invoker.invoke_process(windowed_value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process windowed_value, additional_args, additional_kwargs, output_processor) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window side_inputs = [si[window] for si in self.side_inputs] File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__ _FilteringIterable(self._iterable, target_window), self._view_options) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable return options['data'].view_fn(it) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda> lambda iterable: from_runtime_iterable(iterable, view_options)) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable len(head), str(head[0]), str(head[1]))) ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter'] ERROR:root:Giving up after 4 attempts. WARNING:root:A task failed with exception: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter'] INFO:oauth2client.transport:Attempting refresh to obtain initial access_token INFO:oauth2client.client:Refreshing access_token Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py", line 399, in <module> run() File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py", line 393, in run }, options.view_as(GoogleCloudOptions).project)) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 426, in __exit__ self.run().wait_until_finish() File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 430, in wait_until_finish self._executor.await_completion() File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 400, in await_completion self._executor.await_completion() File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 446, in await_completion raise_(t, v, tb) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call finish_state) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call evaluator.process_element(value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element self.runner.process(element) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process self._reraise_augmented(exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented raise_with_traceback(new_exn) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process return self.do_fn_invoker.invoke_process(windowed_value) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process windowed_value, additional_args, additional_kwargs, output_processor) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window side_inputs = [si[window] for si in self.side_inputs] File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__ _FilteringIterable(self._iterable, target_window), self._view_options) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable return options['data'].view_fn(it) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda> lambda iterable: from_runtime_iterable(iterable, view_options)) File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable len(head), str(head[0]), str(head[1]))) ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']