[FLINK-28786] Cannot run PyFlink 1.16 on MacOS with M1 chip

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.16.2, 1.17.1
• Component/s: API / Python
• Labels: pull-request-available
• Assignee: Huang Xingbo
• Reporter: Ran Tao
• Created: 03/Aug/22 06:48
• Updated: 05/May/23 02:26
• Resolved: 27/Apr/23 10:04

Description

I have tested it with 2 m1 machines. i will reproduce the bug 100%.

1.m1 machine
macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
1.m1 machine
macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9

reproduce step:
1.python -m pip install -r flink-python/dev/dev-requirements.txt
2.cd flink-python; python setup.py sdist bdist_wheel; cd apache-flink-libraries; python setup.py sdist; cd ..;
3.python -m pip install apache-flink-libraries/dist/*.tar.gz
4.python -m pip install dist/*.whl

when run word_count.py it will cause

<frozen importlib._bootstrap>:219: RuntimeWarning: apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate binary incompatibility. Expected 24 from C header, got 32 from PyObject
Traceback (most recent call last):
  File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py", line 129, in <module>
    word_count(known_args.input, known_args.output)
  File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py", line 49, in word_count
    t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py", line 121, in create
    return TableEnvironment(j_tenv)
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py", line 100, in __init__
    self._open()
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py", line 1637, in _open
    startup_loopback_server()
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py", line 1628, in startup_loopback_server
    from pyflink.fn_execution.beam.beam_worker_pool_service import \
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py", line 44, in <module>
    from pyflink.fn_execution.beam import beam_sdk_worker_main  # noqa # pylint: disable=unused-import
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py", line 21, in <module>
    import pyflink.fn_execution.beam.beam_operations # noqa # pylint: disable=unused-import
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py", line 27, in <module>
    from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend, RemoteOperatorStateBackend
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py", line 33, in <module>
    from pyflink.fn_execution.beam.beam_coders import FlinkCoder
  File "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py", line 27, in <module>
    from pyflink.fn_execution.beam import beam_coder_impl_fast as beam_coder_impl
  File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init pyflink.fn_execution.beam.beam_coder_impl_fast
KeyError: '__pyx_vtable__'

Issue Links

• fixes: FLINK-29796 pyflink protobuf requirement out of date
• is duplicated by: FLINK-31968 [PyFlink] 1.17.0 version on M1 processor
• links to: GitHub Pull Request #21341
• links to: GitHub Pull Request #21926
• links to: GitHub Pull Request #22490