Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6537

Limit the batch size for buffering operators based on how much memory they get

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      we are using 16MB for output batch size for all operators by default. however, for buffering operators, depending upon how much memory they get allocated, it is possible that 16MB might be too much to use for output batch size, causing them to spill sometimes.

      Have output batch size to be minimum of 16 MB or 20% of the allocated memory.

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ppenumarthy Padma Penumarthy
            ppenumarthy Padma Penumarthy
            Boaz Ben-Zvi Boaz Ben-Zvi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment