Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-7087

Integrate Arrow's Gandiva into Drill

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      It's a prior work to integrate arrow into drill by invoking the its gandiva feature. Comparing arrow and drill 's in memory column representation , there's different null representation internal now. Drill use 1 byte while arrow using 1 bit to indicate one null row. Also all columns of arrow is nullable now. Apart from those basic differences , they have same memory representation to the different data types.

      The integrating strategy is to invoke arrow's JniWrapper's native method directly by passing the ValueVector's memory address.

      I have done a implementation at our own Drill version by integrating gandiva into Drill's project operator. The performance shows that there's nearly 1 times performance gain at expression computation.

      So if there's no objection , I will submit a related PR to contribute this feature. Also this issue waits for arrow's related issueARROW-4819.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            weijie Weijie Tong
            weijie Weijie Tong

            Dates

              Created:
              Updated:

              Slack

                Issue deployment