[BEAM-12920] Convert To Bare Types Fails on Generators in Python 3.9

• Type: Sub-task
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.32.0
• Fix Version/s: 2.37.0
• Component/s: sdk-py-core
• Environment: Python 3.9.7 ; Linux Mint 20.2, Kernel: 5.4.0-84-generic
• Assignee: Valentyn Tymofieiev
• Reporter: Jonathan Hourany
• Created: 20/Sep/21 23:11
• Updated: 09/Feb/22 22:33
• Resolved: 09/Feb/22 22:33

Description

In Python 3.9 the test test_convert_bare_types fails when testing the following Generator test case

test_cases += [
          (
              'bare generator',
              typing.Generator,
              typehints.Generator[typehints.TypeVariable('T_co')]),
      ]

Raising the following ValueError

./sdks/python/apache_beam/typehints/native_type_compatibility_test.py::NativeTypeCompatibilityTest::test_convert_bare_types Failed: [undefined]ValueError: Unsupported Generator with no arguments.

---

The immediate cause may be stemming from _get_args which attempts to access a type's __args__ attribute. In Python 3.9 bare generators no longer instantiate with a default __args__ attribute. Only when instantiated with args do they get one (see example below).

However,  despite it's name the test doesn't create a generator that's totally bare, so the deeper cause may be coming from the way typinghints.Generator is constructed.

Examples

# Python 3.7.5
In [1]: from typing import Generator

In [2]: Generator.__args__
Out[2]: (+T_co, -T_contra, +V_co)

---

# Python 3.9.7
In [1]: from typing import Generator

In [2]: Generator.__args__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-3e272b948083> in <module>
----> 1 Generator.__args__

~/.pyenv/versions/3.9.7/lib/python3.9/typing.py in __getattr__(self, attr)
    704         if '__origin__' in self.__dict__ and not _is_dunder(attr):
    705             return getattr(self.__origin__, attr)
--> 706         raise AttributeError(attr)
    707 
    708     def __setattr__(self, attr, val):

AttributeError: __args__

In [3]: # Initializing with args, however, creates the needed attr
In [4]: Generator[int, int, int].__args__
Out[4]: (int, int, int)

Issue Links

• blocks: BEAM-12000 Support Python 3.9 in Apache Beam
• links to: GitHub Pull Request #16791