[ARROW-6202] [Java] Exception in thread "main" org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of size 4 due to memory limit. Current allocation: 2147483646

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.1
• Fix Version/s: 0.15.0
• Component/s: Java
• Labels: jdbc pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/22592
• Assignee: Micah Kornfield
• Reporter: Jim Northrup
• Created: 11/Aug/19 22:19
• Updated: 11/Jan/23 07:45
• Resolved: 23/Aug/19 17:29

Description

jdbc query results exceed native heap when using generous -Xmx settings.

for roughly 800 megabytes of csv/flatfile resultset, arrow is unable to house the contents in RAM long enough to persist to disk, without explicit knowledge beyond unit test sample code.

source:
https://github.com/jnorthrup/jdbc2json/blob/master/src/main/java/com/fnreport/QueryToFeather.kt#L83

Exception in thread "main" org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of size 4 due to memory limit. Current allocation: 2147483646
        at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:307)
        at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:277)
        at org.apache.arrow.adapter.jdbc.JdbcToArrowUtils.updateVector(JdbcToArrowUtils.java:610)
        at org.apache.arrow.adapter.jdbc.JdbcToArrowUtils.jdbcToFieldVector(JdbcToArrowUtils.java:462)
        at org.apache.arrow.adapter.jdbc.JdbcToArrowUtils.jdbcToArrowVectors(JdbcToArrowUtils.java:396)
        at org.apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(JdbcToArrow.java:225)
        at org.apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(JdbcToArrow.java:187)
        at org.apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(JdbcToArrow.java:156)
        at com.fnreport.QueryToFeather$Companion.go(QueryToFeather.kt:83)
        at com.fnreport.QueryToFeather$Companion$main$1.invokeSuspend(QueryToFeather.kt:95)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:270)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at com.fnreport.QueryToFeather$Companion.main(QueryToFeather.kt:93)
        at com.fnreport.QueryToFeather.main(QueryToFeather.kt)

Issue Links

• is related to: ARROW-6219 [Java] Add API for JDBC adapter that can convert less then the full result set at a time.
• links to: GitHub Pull Request #5134