[BEAM-12512] Parquetio.py throws "ValueError: invalid literal" when ARROW_MAJOR_VERSION contains alpha-numeric

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Resolved
• Fix Version/s: 2.37.0
• Component/s: io-py-parquet
• Environment: Ubuntu 18.04
• Assignee: Unassigned
• Reporter: Rishi
• Created: 18/Jun/21 16:37
• Updated: 30/Mar/22 21:10
• Resolved: 30/Mar/22 21:10

Description

When Apache Arrow is built from Git branch the resulting version is similar to:

/==================/

/home/arrow/python# python3 setup.py --version
2.0.0.dev0+g478286658.d20210618

/==================/

 This causes exception in apache_beam code at the following line due to presence of alpha-numerics in the generated code:

/==================/

1. python3
   Python 3.6.9 (default, Jan 26 2021, 15:33:00)
   [GCC 8.4.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import apache_beam as beam
   Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/local/lib/python3.6/dist-packages/apache_beam-2.30.0-py3.6-linux-x86_64.egg/apache_beam/_init_.py", line 96, in <module>
   from apache_beam import io
   File "/usr/local/lib/python3.6/dist-packages/apache_beam-2.30.0-py3.6-linux-x86_64.egg/apache_beam/io/_init_.py", line 28, in <module>
   from apache_beam.io.parquetio import *
   File "/usr/local/lib/python3.6/dist-packages/apache_beam-2.30.0-py3.6-linux-x86_64.egg/apache_beam/io/parquetio.py", line 53, in <module>
   ARROW_MAJOR_VERSION, , _ = map(int, pa.version_.split('.'))
   ValueError: invalid literal for int() with base 10: 'dev0+g478286658'

/==================/

Perhaps, determination of ARROW_MAJOR_VERSION can be modified to account for such use cases.

 

Thanks.