Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-525

Scanners use too much memory when reading past scan range

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 1.1
    • Fix Version/s: Impala 1.2.1
    • Component/s: None
    • Labels:
      None

      Description

      The default read-ahead size for scanners is very small, so they end up issuing many small reads past the end of the scan range, each of which uses an 8MB IO buffer. This eventually can cause the process limit to be exceeded and cause queries to fail with "Memory limit exceeded".

      This problem is especially bad for uncompressed RCFiles, which have large data blocks.

        Attachments

          Activity

            People

            • Assignee:
              skye Skye Wanderman-Milne
              Reporter:
              skye Skye Wanderman-Milne
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: