[ARROW-5200] [Java] Provide light-weight arrow APIs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Java
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21675
• Assignee: Liya Fan
• Reporter: Liya Fan
• Created: 23/Apr/19 07:31
• Updated: 11/Jan/23 07:38
• Resolved: 13/May/19 04:13

Description

We are trying to incorporate Apache Arrow to Apache Flink runtime. We find Arrow an amazing library, which greatly simplifies the support of columnar data format.

However, for many scenarios, we find the performance unacceptable. Our investigation shows the reason is that, there are too many redundant checks and computations in Arrow API.

For example, the following figures shows that in a single call to Float8Vector.get(int) method (this is one of the most frequently used APIs in Flink computation),  there are 20+ method invocations.

 

There are many other APIs with similar problems. We believe that these checks will make sure of the integrity of the program. However, it also impacts performance severely. For our evaluation, the performance may degrade by two or three orders of magnitude slower, compared to access data on heap memory. 

We think at least for some scenarios, we can give the responsibility of integrity check to application owners. If they can be sure all the checks have been passed, we can provide some light-weight APIs and the inherent high performance, to them.

In the light-weight APIs, we only provide minimum checks, or avoid checks at all. The application owner can still develop and debug their code using the original heavy-weight APIs. Once all bugs have been fixed, they can switch to light-weight APIs in their products and enjoy the consequent high performance.

 

Issue Links

• is blocked by: ARROW-5209 [Java] Add performance benchmarks from SQL workloads
• is related to: ARROW-5290 [Java] Provide a flag to enable/disable null-checking in vectors' get methods
• relates to: ARROW-3496 [Java] Add microbenchmark code to Java
• relates to: ARROW-5197 [Java] Improving Arrow Vector Reading performance
• relates to: ARROW-1833 [Java] Add accessor methods for data buffers that skip null checking
• links to: GitHub Pull Request #4212