[ARROW-12506] [Python] Improve modularity of pyarrow codebase to speedup compile time

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/28271
• Assignee: Alessandro Molina
• Reporter: Alessandro Molina
• Created: 22/Apr/21 13:52
• Updated: 11/Jan/23 08:26
• Resolved: 27/Apr/21 16:21

Description

There are some modules in pyarrow that end up being fairly big to compile because they are mostly based on including other pxi / pxd files.

That means that when a change to those files is done a big module has to be recompiled slowing down the development process when experimenting (seems it's not uncommon that when a change is done it takes less time to recompile libarrow than pyarrow)

It would be convenient to divide those into separate modules that can lead to separate object files which would allow the compiler to recompile smaller chunks at the time, so that when a change is done we don't have to recompile the whole `lib.pyx` but can just recompile the module where the change is isolated to.

The goal is to allow faster iteration over pyarrow by reducing time spent on waiting for cython compilation on each change.

Issue Links

• links to: GitHub Pull Request #10131
• links to: GitHub Pull Request #10159
• links to: GitHub Pull Request #10162