Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-4714

[C++][Java] Providing JNI interface to Read ORC file via Arrow C++

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Currently if we want to read data from ORC data into Arrow Record Batch in Java runtime, we needs to first use the ORC Java reader to load data into memory then convert it to Arrow RecordBatch.

      However, since ORC Java Reader only read orc data into on heap memory, while Arrow Record only support off heap memory on Java, memory copy in unavoidable in conversion process.  In our internal benchmark, the conversion time can take up to 25% E2E latency when running tpch q1.

      To workaround this overhead, a method is to enable the Java runtime directly reading data from native ORC c++ reader, which will load data directly into off heap memory and only pointer manipulation and schema ser/de would be involved in the conversion process. 

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Yurui Zhou Yurui Zhou Assign to me
            Yurui Zhou Yurui Zhou
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 19h 50m
                19h 50m

                Slack

                  Issue deployment