Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-6776

[Python] Need a lite version of pyarrow



    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.1
    • Fix Version/s: 1.0.0
    • Component/s: Python
    • Labels:


      Currently I am building a library packages on top of pyarrow, so I include pyarrow as a dependency and ship it to our customer. However, when our customer installed our packages, it will also install pyarrow and pyarrow's dependency (numpy). However the dependency size is huge. 

      (py36env) [hyu@c6x64-hyu-newuser-final-clone connector]$ ls -l --block-size=M /home/hyu/py36env/lib/python3.6/site-packages/pyarrow/ 
      total 186M

       And numpy is around 80MB. Total is more than 250 MB.

      Our customer want to bundle all dependency and run the code inside AWS Lambda, however they hit the size limit and failed to run the code.

      Looking into the pyarrow, I saw multiple .so files are shipped both with and without version suffix, I wonder if you can remove the one of them (either with or without suffix), it will at least reduce the package size by half.

      Further, our library just want to use IPC and read data as record batch, I don't need arrow flight at all (which is the biggest .so file and takes around 100 MB). I wonder if you can push a lite version of the pyarrow so that I can specify lite version as the dependency. Or maybe I need to build my own lite version and push it pypi. However, this approach cause further problem if our customer is using the "fat" version of pyarrow unless you the change the namespace of lite version of pyarrow.

      Another alternative is that I bundle the pyarrow with our library ( copy the whole directory into vendored namespace) and ship it to our customer without specifying pyarrow as a dependency. The advantage of this one is that I can build pyarrow with whatever option/sub-module/libraries I need. However, I tried a lot but failed because pyarrow use absolute import and it will fail to import the script in the new location. 

      Any insight how I should resolve this issue?




            • Assignee:
              wesm Wes McKinney
              howryu Haowei Yu
            • Votes:
              0 Vote for this issue
              3 Start watching this issue


              • Created: