Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Impala 2.7.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:
      None

      Description

      To simplify memory transfer, we should change how BufferedTupleStreams are transferred to row batches. Instead of attaching the whole stream object, we should only attach pinned blocks from the stream. This is essentially what the transfer of the stream accomplished.

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-4023: don't attach buffered tuple streams to batches

        This simplifies the memory transfer model by eliminating one category of
        resources that can be attached.

        This patch also separates the concepts of attaching resources and
        flushing resources. Previously RowBatch::AddTupleStream() implicitly
        flushed resources from the ExecNode pipeline, which various ExecNodes
        relied on to free up memory reservations for subsequent processing. In a
        subsequent patch I want the FlushResources() API to become stronger: it
        will force streaming ExecNodes to flush their batches or forces
        blocking ExecNodes to acquire ownership of the memory resources.
        We can't do this right now since we don't have a way to transfer
        ownership of BufferedBlockMgr Blocks.

        Change-Id: I6471422f86ce71e4c6ab277a276000051bc2e8ff
        Reviewed-on: http://gerrit.cloudera.org:8080/4448
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-4023 : don't attach buffered tuple streams to batches This simplifies the memory transfer model by eliminating one category of resources that can be attached. This patch also separates the concepts of attaching resources and flushing resources. Previously RowBatch::AddTupleStream() implicitly flushed resources from the ExecNode pipeline, which various ExecNodes relied on to free up memory reservations for subsequent processing. In a subsequent patch I want the FlushResources() API to become stronger: it will force streaming ExecNodes to flush their batches or forces blocking ExecNodes to acquire ownership of the memory resources. We can't do this right now since we don't have a way to transfer ownership of BufferedBlockMgr Blocks. Change-Id: I6471422f86ce71e4c6ab277a276000051bc2e8ff Reviewed-on: http://gerrit.cloudera.org:8080/4448 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins —

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development