[BEAM-14547] Bug in type inference for Python 3.10

• Type: Bug
• Status: Open
• Priority: P2
• Resolution: Unresolved
• Component/s: sdk-py-core
• Assignee: Anand Inguva
• Reporter: Anand Inguva
• Created: 01/Jun/22 22:21
• Updated: 05/Jun/22 01:14

Description

After debugging, the issue arises here

https://github.com/apache/beam/blob/70cee1d125db0b62b2d660fc399e5a23845eb0e3/sdks/python/apache_beam/typehints/trivial_inference.py#L328

The method above is not able to infer outputs properly in Python 3.10 and results in an outputting wrong output type. 

# Python <= 3.9 

from apache_beam.typehints.trivial_inference import infer_return_type_func  

fn = lambda x: False
wrapper = lambda x, *args, **kwargs: [x] if fn(x, *args, **kwargs) else []
output_type = infer_return_type_func(wrapper, [int])
# output_type would be List[int]

# Python == 3.10

output_type = infer_return_type_func(wrapper, [int]) 
# fails with error beam/sdks/python/apache_beam/typehints/opcodes.py", line # 322, in dict_update base = state.stack[-arg]
# IndexError: list index out of range

# in python 3.9
from dis import dis
print(dis(wrapper))

'''
1           0 LOAD_GLOBAL              0 (fn)
              2 LOAD_FAST                0 (x)
              4 BUILD_LIST               1
              6 LOAD_FAST                1 (args)
              8 LIST_EXTEND              1
             10 LIST_TO_TUPLE
             12 BUILD_MAP                0
             14 LOAD_FAST                2 (kwargs)
             16 DICT_MERGE               1
             18 CALL_FUNCTION_EX         1
             20 POP_JUMP_IF_FALSE       28
             22 LOAD_FAST                0 (x)
             24 BUILD_LIST               1
             26 RETURN_VALUE
        >>   28 BUILD_LIST               0
             30 RETURN_VALUE
'''

# in python 3.10 

print(dis(wrapper))
'''
1           0 LOAD_GLOBAL              0 (fn)
              2 LOAD_FAST                0 (x)
              4 BUILD_LIST               1
              6 LOAD_FAST                1 (args)
              8 LIST_EXTEND              1
             10 LIST_TO_TUPLE
             12 BUILD_MAP                0
             14 LOAD_FAST                2 (kwargs)
             16 DICT_MERGE               1
             18 CALL_FUNCTION_EX         1
             20 POP_JUMP_IF_FALSE       14 (to 28)
             22 LOAD_FAST                0 (x)
             24 BUILD_LIST               1
             26 RETURN_VALUE
        >>   28 BUILD_LIST               0
             30 RETURN_VALUE
'''

Issue Links

• blocks: BEAM-13590 Changes for Python 3.10
• is related to: BEAM-12914 Support LIST_EXTEND and LIST_TO_TUPLE opcodes in type inference on Python 3.9