Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-809

TableScan does not support large/infinite scans

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: core
    • Labels:
      None

      Description

      When running a simple query (e.g. select stream * from orders) and the StreamableTable returning a Enumerable<Object[]> backed by an infinite stream, you end up quickly trying to store all the values of the stream in a ListSink in a TableScanNode.

      From Julian:

      You're hitting the interpreter "cheap and dirty"
      implementation of TableScan. I made the interpreter the simplest thing
      that could possibly work, so I made every operator build a list. (I
      know, I know. Enumerable uses iterators, and other implementations do
      even better. But I wanted to fit it into one page of code.)

      ...

      The solution will be either to fix the interpreter to use iterators
      (or similar) rather than lists, or to recognize that a query is
      infinite and not use the interpreter.

        Attachments

        1. calcite-803-v0.patch
          20 kB
          Jesse Yates
        2. calcite-803-v1.patch
          24 kB
          Jesse Yates

          Issue Links

            Activity

              People

              • Assignee:
                julianhyde Julian Hyde
                Reporter:
                jesse_yates Jesse Yates
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: