[BEAM-12914] Support LIST_EXTEND and LIST_TO_TUPLE opcodes in type inference on Python 3.9

• Type: Sub-task
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Affects Version/s: 2.32.0
• Fix Version/s: 2.37.0
• Component/s: sdk-py-core
• Environment: Python 3.9.7 ; Linux Mint 20.2, Kernel: 5.4.0-84-generic

Language

Python

• Assignee: Robert Bradshaw
• Reporter: Jonathan Hourany
• Created: 18/Sep/21 23:51
• Updated: 01/Jun/22 22:49
• Resolved: 09/Feb/22 19:23

Description

The order and/or type of opcodes used to build lambda functions have changed in Python 3.9 (see example below). This causes the any test that relies on `trivial_inference.infer_return_type` to fail because the function returns a default of `Any` in cases where the type should normally be inferable. Tests cases that fail include:

• testBuildListUnpack 
• testBuildTupleUnpack
• testBuildTupleUnpackWithCall
• test_pardo_type_inference

Test failure messages are all variations of:

./sdks/python/apache_beam/transforms/ptransform_test.py::PTransformTypeCheckTestCase::test_pardo_type_inference Failed: [undefined]AssertionError: <class 'int'> != Any
./sdks/python/apache_beam/typehints/trivial_inference_test.py::TrivialInferenceTest::testBuildListUnpack Failed: [undefined]AssertionError: List[int] != Any

etc

Example

# Python3.7

In [1]: from dis import dis

In [2]: from apache_beam.typehints import typehints

In [3]: beam_type_hints = [typehints.List[int]]

In [4]: dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
  1           0 LOAD_FAST                0 (_list)
              2 LOAD_FAST                0 (_list)
              4 LOAD_FAST                0 (_list)
              6 BUILD_LIST_UNPACK        3
              8 LOAD_GLOBAL              0 (beam_type_hints)
             10 CALL_FUNCTION            1
             12 RETURN_VALUE

---

Whereas in Python 3.9

# Python 3.9
In [1]: from dis import dis

In [2]: from apache_beam.typehints import typehints

In [3]: beam_type_hints = [typehints.List[int]]

In [4]: dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
<>:1: SyntaxWarning: 'list' object is not callable; perhaps you missed a comma?
<ipython-input-4-6499114c3d61>:1: SyntaxWarning: 'list' object is not callable; perhaps you missed a comma?
  dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
  1           0 BUILD_LIST               0
              2 LOAD_FAST                0 (_list)
              4 LIST_EXTEND              1
              6 LOAD_FAST                0 (_list)
              8 LIST_EXTEND              1
             10 LOAD_FAST                0 (_list)
             12 LIST_EXTEND              1
             14 LOAD_GLOBAL              0 (beam_type_hints)
             16 CALL_FUNCTION            1
             18 RETURN_VALUE

The `SyntaxWarning` is most likely raising because enhancment in bpo-15248 misses the fact that the lambda expression is a function, not a list we're calling, and isn't relevant to the issue.

 

Issue Links

• blocks: BEAM-12000 Support Python 3.9 in Apache Beam
• is related to: BEAM-13272 Support type inference on Python 3.9
• relates to: BEAM-14547 Bug in type inference for Python 3.10
• links to: GitHub Pull Request #16761