[BEAM-1251] Python 3 Support

• Type: Improvement
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.11.0
• Component/s: sdk-py-core
• Assignee: Valentyn Tymofieiev
• Reporter: Eyad Sibai
• Created: 08/Jan/17 11:23
• Updated: 13/Dec/23 17:54
• Resolved: 28/Oct/19 18:26

Description

FAQ

Does Apache Beam support Python 3?

• Yes!

Is there any remaining work?

• We continue to improve user experience of Python 3 users, add support for new Python minor versions, and phase out support of old ones.
  Check out the Python SDK roadmap on how to contribute or report a Python 3 issue!

Which SDK version should I use?

• For best experience, use the latest released SDK. For summary of Py3-related changes, read this thread.

Help! I am getting a pickling error in StockUnpickler.find_class() on Python 3.

• Does the error happens in load_session call? See BEAM-6158 .
• Do you use Beam SDK less than 2.17.0? See BEAM-8651.

My Avro Sink no longer works.

• Beam switched to use FastAvro as a default library on Python 3. The fastavro-based Avro sink expects schema as a dictionary, while the avro-python3-based Avro Sink expects a schema that was previously parsed by avro.schema.Parse(). Fastavro will not accept a schema parsed by avro-python3, so make sure you pass the correct schema. See: BEAM-10769.

My streaming pipelines are stuck on Python 3.

• Do you use Beam SDK less than 2.17.0? If so please upgrade to 2.17.0. See BEAM-8651.

Issue Links

• duplicates: BEAM-1373 Update Python SDK code to support both Python 2 and 3
• incorporates: BEAM-4000 Futurize and fix python 2 compatibility for io subpackage
• incorporates: BEAM-4002 Futurize and fix python 2 compatibility for options subpackage
• incorporates: BEAM-4004 Futurize and fix python 2 compatibility for testing subpackage
• incorporates: BEAM-4005 Futurize and fix python 2 compatibility for tools subpackage
• incorporates: BEAM-4006 Futurize and fix python 2 compatibility for transforms subpackage
• incorporates: BEAM-4009 Futurize and fix python 2 compatibility for unpackaged files
• incorporates: BEAM-4511 Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.
• incorporates: BEAM-3381 Handle trivial inference with default arguments
• incorporates: BEAM-3730 typehints.TypeVariable issues with __hash__
• incorporates: BEAM-4229 PostCommits broken on SpannerReadIT tests
• incorporates: BEAM-1368 Python setup should be portable to Python 3.x
• incorporates: BEAM-3671 Need Python 3.x interpreter in Jenkins machines
• incorporates: BEAM-1373 Update Python SDK code to support both Python 2 and 3
• incorporates: BEAM-2784 Fix issues from automated conversion to allow Python 2 functionality
• incorporates: BEAM-2786 Update jenkins test scripts to test with Py2 & Py3
• incorporates: BEAM-3233 Fix Py3 list comprehension type inferance
• incorporates: BEAM-3761 Fix Python 3 cmp function
• incorporates: BEAM-4715 Have Beam tests running and passing on Python 3
• incorporates: BEAM-3380 Make type inference compatible with Python 3.6
• incorporates: BEAM-3781 Figure out min supported Python 3 version
• is a parent of: BEAM-4511 Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.
• is a parent of: BEAM-6877 TypeHints Py3 Error: Type inference tests fail on Python 3.6 due to bytecode changes
• is a parent of: BEAM-5720 Default coder breaks with large ints on Python 3
• is a parent of: BEAM-5255 Fix over-aggressive division futurization.
• is a parent of: BEAM-4543 Remove dependency on googledatastore in favor of google-cloud-datastore.
• is a parent of: BEAM-6983 Python 3.6 Support
• is a parent of: BEAM-6984 Python 3.7 Support
• is a parent of: BEAM-8494 Python 3.8 Support
• is blocked by: BEAM-2784 Fix issues from automated conversion to allow Python 2 functionality
• is related to: BEAM-8441 Python 3 pipeline fails with errors in StockUnpickler.find_class() during loading a main session.
• links to: GitHub Pull Request #4570
• links to: GitHub Pull Request #4571
• links to: GitHub Pull Request #4697
• links to: GitHub Pull Request #4804
• links to: GitHub Pull Request #4820
• links to: GitHub Pull Request #4834
• links to: GitHub Pull Request #4835
• links to: GitHub Pull Request #5531
• links to: GitHub Pull Request #5842
• links to: GitHub Pull Request #5869
• links to: GitHub Pull Request #5887
• links to: GitHub Pull Request #5890
• links to: GitHub Pull Request #5908
• links to: GitHub Pull Request #5911
• links to: GitHub Pull Request #6166
• links to: GitHub Pull Request #6489
• links to: GitHub Pull Request #6650
• links to: GitHub Pull Request #6678
• links to: GitHub Pull Request #6679
• links to: GitHub Pull Request #7446
• links to: GitHub Pull Request #7953
• links to: GitHub Pull Request #8956
• links to: GitHub Pull Request #9537
• links to: GitHub Pull Request #14198

Sub-Tasks

• Support Python native types in Beam typehints: Resolved
• Make the coders package compatible with Python 3: Resolved
• Enable tests to run in Python 3: Resolved
• Finish io futurize stage 2: fix the missing pylint3 check in tox.ini: Resolved
• Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.: Resolved
• Add an SDK harness container with Python 3 interpreter for portable pipelines.: Resolved
• Exercise Python 3 SDK harness container in ValidatesContainer Jenkins test suite.: Resolved
• Finish Python 3 porting for coders module: Resolved
• Finish Python 3 porting for examples module: Resolved
• Finish Python 3 porting for internal module: Resolved
• Finish Python 3 porting for io module: Resolved
• Finish Python 3 porting for metrics module: Resolved
• Finish Python 3 porting for options module: Resolved
• Finish Python 3 porting for portability module: Resolved
• Finish Python 3 porting for runners module: Resolved
• Finish Python 3 porting for testing module: Resolved
• Finish Python 3 porting for transforms module: Resolved
• Finish Python 3 porting for typehints module: Resolved
• Finish Python 3 porting for utils module: Resolved
• Finish Python 3 porting for unpackaged files: Resolved
• Add tox suites to exercise unit tests using Python3 interpreter with cython, and with gcp dependencies.: Resolved
• Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function: Resolved
• Several tests fail on Python 3 with Failed assert: [<some number>] == [nan]: Resolved
• Side inputs don't work on Python 3: Resolved
• Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' and 'EmptySideInput': Resolved
• Some tests use assertItemsEqual method, not available in Python 3: Resolved
• Several tests fail on Python 3 with TypeError: unorderable types: str() < int(): Resolved
• Several tests fail on Python 3 with: Runtime type violation detected: Resolved
• Several IO tests hang indefinitely during execution on Python 3.: Resolved
• Avro IO does not work with avro-python3 package out-of-the-box on Python 3, several tests fail with AttributeError (module 'avro.schema' has no attribute 'parse'): Resolved
• Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}: Resolved
• Redesign test_split_at_fraction_exhaustive tests for Python 3: Open
• VcfIO is not Python3-compatible and there are no plans to make it compatible.: Resolved
• Several typehints tests fail on Python 3 with ValueError: no signature found for builtin <method 'upper' of 'str' objects>: Resolved
• Add tox suites for various Python 3 versions (3.5, 3.6, 3.7): Triage Needed
• Default coder breaks with large ints on Python 3: Resolved
• Disable compare parameter in Top.Of() combiner when executing in Python 3.: Resolved
• Util test on annotations fails: Resolved
• Using methods in map is broken on Python 3: Resolved
• Validates runner tests fail with: Cannot convert bytes value to JSON value: Resolved
• wordcount_fnapi_it failed on TestDataflowRunner because of JSON string decoding error: Resolved
• Support DoFns with Keyword-only arguments in Python 3.: Triage Needed
• TFRecordio not Py3 compatible: Resolved
• Enable WordCount example on DataflowRunner on Python 3: Resolved
• Gradle setupVirtualenv supports Python 3: Resolved
• Revert dill pip install from github commit: Resolved
• Gcsio batch delete broken in Python 3: Resolved
• Using --save_main_session fails on Python 3 when main module has invocations of superclass method using 'super' .: Triage Needed
• Opcounters sampling test fails for some random seeds on Python3: Resolved
• TypeError in DataflowRunner: dict_values does not support indexing: Resolved
• Dill fails to pickle avro.RecordSchema classes on Python 3.: Resolved
• Parallel tox (unit) tests run on Jenkins: Resolved
• BigQuery IO does not work in Python 3: Resolved
• TypeHints Py3 Error: TrivialInferenceTest.testTupleListComprehension fails on Python 3: Resolved
• GCS IO tests are very flaky under Python 3.5: Resolved
• Dataflow Python runner should use a Python-3 compatible container when starting a Python 3 pipeline.: Resolved
• Add integration test on DirectRunner in Python 3: Resolved
• Beam Python SDK release qualification should verify supported Python 3 versions.: Resolved
• Stager should stage Python 3 wheels for Beam SDK once they are released.: Resolved
• Release Python 3 wheels with first Beam SDK release that supports Python 3.: Resolved
• Add PostCommit suite for integration tests on DataflowRunner: Resolved
• Exercise Dataflow runner integration tests in a postcommit suite for Python 3.5 and 3.6: Resolved
• Dataflow ValidatesRunner test suite should also exercise ValidatesRunner tests under Python 3.: Resolved
• Exercise direct runner integration tests in a postcommit suite for Python 3.5 and 3.6.: Resolved
• SDK source tarball is different when created on Python 2 and Python 3: Resolved
• Typehinting depends on typing changes in Python 3.5.3: Resolved
• Bigquery Tornadoes IT is broken in Python3 PostCommit test suite.: Resolved
• Block size difference in avro library on Python3 causes some AvroIO tests to fail.: Resolved
• BigQuery IO does not support bytes in Python 3: Triage Needed
• Add Streaming wordcount test to Dataflow ValidatesContainer test suite: Resolved
• python 3 test_hourly_team_score_it fails with bigquery job id already exists: Resolved
• test_multimap_side_input in fn_api.runner_test fails on Python 3.6: Resolved
• Add Python3 performance benchmarks: Resolved
• Configurable Python interpreter version in Gradle: Resolved
• Design Py3-compatible typehints annotation support in Beam 3.: Triage Needed
• Enable use_fastavro experiment on Dataflow Runner for all Py3 jobs.: Resolved
• Add DirectRunnerIT test suite to Python3 Postcommit suite.: Resolved
• TypeError caused by using str variable as header argument in apache_beam.io.textio.WriteToText: Resolved
• Rename ToStringCoder into ToBytesCoder: Resolved
• Dataflow runner should set use_fastavro experiment on Python 3.: Resolved
• Add ValidatesRunner test suite for Flink on Python 3.: Resolved
• Enable Python3 tests for Spark: Resolved
• Support Py3 Dataclasses: Resolved
• Revise BQ integration tests to clearly communicate that BQ IO expects base64-encoded bytes.: Resolved
• apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow: Resolved
• Clean up Python 2 codepaths once Beam no longer supports Python 2.: Triage Needed
• FastAvroTest has slow test_dynamic_exhaustive on Python 2 and 3.: Resolved
• Create a Wordcount-on-Flink Python 3 test suite.: Resolved
• Document Python 3 support in Beam starting from 2.14.0: Resolved
• Add Python 3.6, 3.7 as supported qualifiers to setup.py.: Resolved
• Improve Avro IO integration test coverage on Python 3.: Resolved
• Add smoke integration tests to Precommit test suites on Python 3: Resolved
• Add SDK harness containers for Py 3.6, Py 3.7: Resolved
• deadlock using save_main_session and logging caused by threading.RLock pickling: Open
• Add integration tests for HDFS: Resolved
• Add ITs to check IO behavior with bytes and unicode strings: Resolved
• Accept Py3 wheels in SDK harness container.: Resolved
• Unify test suite configuration structure across Py2 and Py 3 suites: Resolved
• Python 3 test parallelization causes test flakines due to ModuleNotFoundError.: Resolved
• Implement support of PEP 484 annotations for user functions in transforms such as ParDo, Combine in Py3.: Resolved
• Migrate to "typing" module typing types in Beam typehints (on Py2 and Py3).: Resolved
• Allow retries of PostCommit test suites per Python version: Resolved
• Use a Python3-compatible profiler in apache_beam.utils.profiler: Resolved
• Add key type conversion in from and to client entity in Datastore v1new IO.: Resolved
• Add Python 2 deprecation warnings starting from 2.17.0 release.: Resolved
• Generate Python SDK docs using Python 3: Resolved
• UserScore example fails on Python 3.5 as of 2.13.0 and 2.14.0 with Dataflow runner: Resolved
• Run pylint in Python 3: Resolved
• Add a Python 3 test scenario for MongoDB IO: Resolved
• --profile_memory flag is py2 only: Resolved
• Provide a way to better control minor+patch versions of Python 3.x interpreters used to run Beam tests locally and on Jenkins.: Open
• Update documentation for Python 3 support after Beam 2.16.0.: Resolved
• Establish consensus around how many concurrent minor versions of Python Beam should support, and deprecation policy for older versions.: Resolved
• Python 3 portable pipelines sometimes fail with errors in StockUnpickler.find_class(): Triage Needed
• Drop support for Python 3.5: Triage Needed
• Create a page that describes what it takes to add and sunset support of a Python minor version in Beam.: Open
• Mark 2.24.0 as the last release supporting Python 2 in release notes and warnings.: Triage Needed
• python3.9.1 support: Resolved