[BEAM-5457] BigQuerySource(query=...) in DirectRunner creates temp dataset in the wrong location

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0
• Component/s: sdk-py-core
• Assignee: Chamikara Madhusanka Jayalath
• Reporter: Joar Wandborg
• Created: 21/Sep/18 09:46
• Updated: 16/May/20 13:59
• Resolved: 08/Oct/18 23:52

Description

I'm in the EU, if I have a

 

BigQuerySource(
    query="SELECT x, y FROM `my-other-project.mydataset.my_european_table`",
    project="myproject",
    use_standard_sql=True
)

And then run the Pipeline through the DirectRunner I get the following warning and error:

2018-09-21 11:39:52,620 WARNING root create_temporary_dataset
Dataset myproject:temp_dataset_0bbb28f014a24225b668a67341f4f71e does not exist so we will create it as temporary with location=None 

HttpBadRequestError: HttpError accessing <https://www.googleapis.com/bigquery/v2/projects/myproject/queries/xyz123?alt=json&maxResults=10000>: response: <{'status': '400', 'content-length': '354', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Fri, 21 Sep 2018 09:39:55 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443"; ma=2592000; v="44,43,39,35"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
  "error": {
    "code": 400,
    "message": "Cannot read and write in different locations: source: EU, destination: US",
    "errors": [
      {
        "message": "Cannot read and write in different locations: source: EU, destination: US",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }

There's a TODO in the code that looks very related: https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665

 

Issue Links

• relates to: BEAM-1909 BigQuery read transform fails for DirectRunner when querying non-US regions
• links to: GitHub Pull Request #6463