Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-809

TableScan does not support large/infinite scans

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0
    • core
    • None

    Description

      When running a simple query (e.g. select stream * from orders) and the StreamableTable returning a Enumerable<Object[]> backed by an infinite stream, you end up quickly trying to store all the values of the stream in a ListSink in a TableScanNode.

      From Julian:

      You're hitting the interpreter "cheap and dirty"
      implementation of TableScan. I made the interpreter the simplest thing
      that could possibly work, so I made every operator build a list. (I
      know, I know. Enumerable uses iterators, and other implementations do
      even better. But I wanted to fit it into one page of code.)

      ...

      The solution will be either to fix the interpreter to use iterators
      (or similar) rather than lists, or to recognize that a query is
      infinite and not use the interpreter.

      Attachments

        1. calcite-803-v1.patch
          24 kB
          Jesse Yates
        2. calcite-803-v0.patch
          20 kB
          Jesse Yates

        Issue Links

          Activity

            People

              julianhyde Julian Hyde
              jesse_yates Jesse Yates
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: