[BEAM-2101] Reading public GCS files requires authentication

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: sdk-py-core
• Assignee: Udi Meiri
• Reporter: Ahmet Altay
• Created: 27/Apr/17 17:07
• Updated: 16/May/20 13:58
• Resolved: 18/Dec/17 20:04

Description

cc: sb2nov

[e-mail] reported this on hackathon, while running wordcount. Even though plain gsutil works fine on the public file, wordcount fails. It is possible that some other authentication is required as part of the validation.

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "apache_beam/io/textio.py", line 390, in _init_
skip_header_lines=skip_header_lines)
File "apache_beam/io/textio.py", line 89, in _init_
validate=validate)
File "apache_beam/io/filebasedsource.py", line 103, in _init_
self._validate()
File "apache_beam/utils/value_provider.py", line 101, in _f
return fnc(self, *args, **kwargs)
File "apache_beam/io/filebasedsource.py", line 163, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File "apache_beam/io/filesystems.py", line 81, in match
return filesystem.match(patterns, limits)
File "apache_beam/io/gcp/gcsfilesystem.py", line 100, in match
raise BeamIOError("Match operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions

{'gs://apache-beam-samples/shakespeare/hamlet.txt': ApplicationDefaultCredentialsError('The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.',)}

Issue Links

• links to: GitHub Pull Request #4254