[FLINK-25188] Cannot install PyFlink on MacOS with M1 chip

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.16.0
• Component/s: API / Python
• Labels: pull-request-available
• Assignee: Luning Wang
• Reporter: Luning Wang
• Created: 06/Dec/21 09:37
• Updated: 07/Jun/22 08:03
• Resolved: 06/Jun/22 06:21

Description

Need to update dependencies: numpy>= 1.20.3、pyarrow>=5.0.0、pandas>=1.3.0、apache-beam==2.36.0

This following is some dependencies adapt M1 chip informations

Numpy version:

https://stackoverflow.com/questions/65336789/numpy-build-fail-in-m1-big-sur-11-1

https://github.com/numpy/numpy/releases/tag/v1.21.4

pyarrow version:

https://stackoverflow.com/questions/68385728/installing-pyarrow-cant-copy-build-lib-macosx-11-arm64-3-9-pyarrow-include-ar

pandas version:

https://github.com/pandas-dev/pandas/issues/40611#issuecomment-901569655

Apache beam:

https://issues.apache.org/jira/browse/BEAM-12957

https://issues.apache.org/jira/browse/BEAM-11703

This following is dependency tree after installed successfully 

Although Beam need numpy<1.21.0 and M1 need numpy >=1.21.4, when I using numpy 1.20.3  I install successfully on M1 chip.

apache-flink==1.14.dev0
  - apache-beam [required: ==2.34.0, installed: 2.34.0]
    - avro-python3 [required: >=1.8.1,<1.10.0,!=1.9.2, installed: 1.9.2.1]
    - crcmod [required: >=1.7,<2.0, installed: 1.7]
    - dill [required: >=0.3.1.1,<0.3.2, installed: 0.3.1.1]
    - fastavro [required: >=0.21.4,<2, installed: 0.23.6]
      - pytz [required: Any, installed: 2021.3]
    - future [required: >=0.18.2,<1.0.0, installed: 0.18.2]
    - grpcio [required: >=1.29.0,<2, installed: 1.42.0]
      - six [required: >=1.5.2, installed: 1.16.0]
    - hdfs [required: >=2.1.0,<3.0.0, installed: 2.6.0]
      - docopt [required: Any, installed: 0.6.2]
      - requests [required: >=2.7.0, installed: 2.26.0]
        - certifi [required: >=2017.4.17, installed: 2021.10.8]
        - charset-normalizer [required: ~=2.0.0, installed: 2.0.9]
        - idna [required: >=2.5,<4, installed: 3.3]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]
      - six [required: >=1.9.0, installed: 1.16.0]
    - httplib2 [required: >=0.8,<0.20.0, installed: 0.19.1]
      - pyparsing [required: >=2.4.2,<3, installed: 2.4.7]
    - numpy [required: >=1.14.3,<1.21.0, installed: 1.20.3]
    - oauth2client [required: >=2.0.1,<5, installed: 4.1.3]
      - httplib2 [required: >=0.9.1, installed: 0.19.1]
        - pyparsing [required: >=2.4.2,<3, installed: 2.4.7]
      - pyasn1 [required: >=0.1.7, installed: 0.4.8]
      - pyasn1-modules [required: >=0.0.5, installed: 0.2.8]
        - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
      - rsa [required: >=3.1.4, installed: 4.8]
        - pyasn1 [required: >=0.1.3, installed: 0.4.8]
      - six [required: >=1.6.1, installed: 1.16.0]
    - orjson [required: <4.0, installed: 3.6.5]
    - protobuf [required: >=3.12.2,<4, installed: 3.17.3]
      - six [required: >=1.9, installed: 1.16.0]
    - pyarrow [required: >=0.15.1,<6.0.0, installed: 5.0.0]
      - numpy [required: >=1.16.6, installed: 1.20.3]
    - pydot [required: >=1.2.0,<2, installed: 1.4.2]
      - pyparsing [required: >=2.1.4, installed: 2.4.7]
    - pymongo [required: >=3.8.0,<4.0.0, installed: 3.12.2]
    - python-dateutil [required: >=2.8.0,<3, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.16.0]
    - pytz [required: >=2018.3, installed: 2021.3]
    - requests [required: >=2.24.0,<3.0.0, installed: 2.26.0]
      - certifi [required: >=2017.4.17, installed: 2021.10.8]
      - charset-normalizer [required: ~=2.0.0, installed: 2.0.9]
      - idna [required: >=2.5,<4, installed: 3.3]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]
    - typing-extensions [required: >=3.7.0,<4, installed: 3.10.0.2]
  - apache-flink-libraries [required: ==1.14.dev0, installed: 1.14.dev0]
  - avro-python3 [required: >=1.8.1,<1.10.0,!=1.9.2, installed: 1.9.2.1]
  - cloudpickle [required: ==1.2.2, installed: 1.2.2]
  - fastavro [required: >=0.21.4,<0.24, installed: 0.23.6]
    - pytz [required: Any, installed: 2021.3]
  - numpy [required: >=1.20.3, installed: 1.20.3]
  - pandas [required: >=1.3.0, installed: 1.3.0]
    - numpy [required: >=1.17.3, installed: 1.20.3]
    - python-dateutil [required: >=2.7.3, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.16.0]
    - pytz [required: >=2017.3, installed: 2021.3]
  - protobuf [required: <3.18, installed: 3.17.3]
    - six [required: >=1.9, installed: 1.16.0]
  - py4j [required: ==0.10.8.1, installed: 0.10.8.1]
  - pyarrow [required: >=5.0.0, installed: 5.0.0]
    - numpy [required: >=1.16.6, installed: 1.20.3]
  - python-dateutil [required: ==2.8.0, installed: 2.8.0]
    - six [required: >=1.5, installed: 1.16.0]
  - pytz [required: >=2018.3, installed: 2021.3]
  - requests [required: >=2.26.0, installed: 2.26.0]
    - certifi [required: >=2017.4.17, installed: 2021.10.8]
    - charset-normalizer [required: ~=2.0.0, installed: 2.0.9]
    - idna [required: >=2.5,<4, installed: 3.3]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7] 

 

 

Issue Links

• Dependent: FLINK-27575 Support PyFlink in Python3.9
• Dependent: FLINK-27058 Add support for Python 3.9
• is related to: FLINK-26070 Update dependency numpy to >=1.21.0,<1.22
• relates to: FLINK-23230 Cannot compile Flink on MacOS with M1 chip
• links to: GitHub Pull Request #18769
• links to: GitHub Pull Request #19802