[BEAM-5624] Avro IO does not work with avro-python3 package out-of-the-box on Python 3, several tests fail with AttributeError (module 'avro.schema' has no attribute 'parse')

• Type: Sub-task
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.9.0
• Component/s: sdk-py-core
• Assignee: Simon
• Reporter: Valentyn Tymofieiev
• Created: 03/Oct/18 03:31
• Updated: 16/May/20 13:54
• Resolved: 12/Oct/18 07:34

Description

======================================================================
ERROR: Failure: AttributeError (module 'avro.schema' has no attribute 'parse')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 693, in _load
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 673, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/avroio_test.py", line 54, in <module>
class TestAvro(unittest.TestCase):
File "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/avroio_test.py", line 89, in TestAvro
SCHEMA = avro.schema.parse('''
AttributeError: module 'avro.schema' has no attribute 'parse'

Note that we use a different implementation of avro/avro-python3 package depending on Python version. We are also evaluating potential replacement of avro with fastavro.

Issue Links

• blocks: BEAM-5315 Finish Python 3 porting for io module
• links to: GitHub Pull Request #6616