Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2758

Remove BufferedTupleStream::GetRows()

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.3.0
    • Fix Version/s: None
    • Component/s: Backend

      Description

      The BufferedTupleStream::GetRows() API returns all of the rows from a pinned stream in a single row batch. This is not a very good API, particularly if we wish to simplify memory management because it can result in creating an arbitrarily large RowBatch. This comes with unbounded memory overhead for tuple pointers, and has potential to cause problems in code that does not anticipate very large batches.

      Instead we should convert callsites to call BufferedTupleStream::GetNext() and process a batch at a time.

        Activity

        Hide
        jbapple Jim Apple added a comment -

        It would be lovely if there was a way to iterate over the rows without the overhead of maintaining the batch state in each caller. Perhaps some sort of smart iterator would do the trick?

        Show
        jbapple Jim Apple added a comment - It would be lovely if there was a way to iterate over the rows without the overhead of maintaining the batch state in each caller. Perhaps some sort of smart iterator would do the trick?
        Hide
        tarmstrong Tim Armstrong added a comment -

        I think something like that would be ideal. I think in the short term we want to minimise the surface area of the BufferedTupleStream API in anticipation of reworking it, so probably it's most important here to do something simple

        Show
        tarmstrong Tim Armstrong added a comment - I think something like that would be ideal. I think in the short term we want to minimise the surface area of the BufferedTupleStream API in anticipation of reworking it, so probably it's most important here to do something simple

          People

          • Assignee:
            tianyiwang Tianyi Wang
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development