Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3134

cannot write data to BigQuery with Dataflow

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: sdk-py-core
    • Labels:
      None

      Description

      (sample code with description is here https://github.com/zdenulo/dataflow_bigquery_error)

      I was running for the first time Dataflow job (with version 2.1.1) to read data from BigQuery, make some modifications, then write data to different table in BigQuery. When I was running locally (on small subset) it was ok, but when I tried to run on Dataflow I get following exception:

      apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
      (ade3180ffa878a6b): Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 706, in run
      self._load_main_session(self.local_staging_directory)
      File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 446, in _load_main_session
      pickler.load_session(session_file)
      File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 247, in load_session
      return dill.load_session(file_path)
      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 363, in load_session
      module = unpickler.load()
      File "/usr/lib/python2.7/pickle.py", line 858, in load
      dispatch[key](self)
      File "/usr/lib/python2.7/pickle.py", line 1182, in load_append
      list.append(value)
      File "/usr/local/lib/python2.7/dist-packages/apitools/base/protorpclite/messages.py", line 1142, in append
      self.__field.validate_element(value)
      AttributeError: 'FieldList' object has no attribute 'FieldList_field'

      In my opinion, it looks like it has to do something with pickling schema definition for output table.

        Attachments

          Activity

            People

            • Assignee:
              altay Ahmet Altay
              Reporter:
              zdenulo Zdenko Hrcek
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: