Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-809

TableScan does not support large/infinite scans

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0
    • core
    • None

    Description

      When running a simple query (e.g. select stream * from orders) and the StreamableTable returning a Enumerable<Object[]> backed by an infinite stream, you end up quickly trying to store all the values of the stream in a ListSink in a TableScanNode.

      From Julian:

      You're hitting the interpreter "cheap and dirty"
      implementation of TableScan. I made the interpreter the simplest thing
      that could possibly work, so I made every operator build a list. (I
      know, I know. Enumerable uses iterators, and other implementations do
      even better. But I wanted to fit it into one page of code.)

      ...

      The solution will be either to fix the interpreter to use iterators
      (or similar) rather than lists, or to recognize that a query is
      infinite and not use the interpreter.

      Attachments

        1. calcite-803-v1.patch
          24 kB
          Jesse Yates
        2. calcite-803-v0.patch
          20 kB
          Jesse Yates

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            julianhyde Julian Hyde
            jesse_yates Jesse Yates
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment