Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2755

Clean up memory management in backend

    Details

      Description

      This is an umbrella task for fixing bugs and simplifying backend code related to memory management in preparation for larger changes to how memory management is done.

        Issue Links

          Issues in Epic

          There are no issues in this epic.

            Activity

            Hide
            tarmstrong Tim Armstrong added a comment -

            commit 0568c9ba56ecbf2d9bda28d495bb548404470f18
            Author: Tim Armstrong <tarmstrong@cloudera.com>
            Date: Thu Nov 19 14:59:32 2015 -0800

            Simplify PAGG ProcessBatch

            The PAGG insert logic can be simplified by exploiting the fact that
            HashTable::Insert will not fail unless a duplicate node is being
            allocated. In PAGG, duplicate nodes are never allocated. Therefore
            the code does not need to check for or handle these failures. As
            a further consequence, no special handling of the current
            intermediate_tuple is needed when spilling a partition.

            This patch also makes the code more readable by factoring the main
            ProcessBatch() function into several smaller functions. The main codegen'd
            inner loops are still inlined into ProcessBatch().

            Also delete some dead code and rearrange some code to hopefully be more
            readable.

            Change-Id: I74398f067fa6398d9800eeec1db3db6fb9235160
            Reviewed-on: http://gerrit.cloudera.org:8080/1509
            Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
            Tested-by: Internal Jenkins

            be/src/exec/hash-table.h | 2 +
            be/src/exec/partitioned-aggregation-node-ir.cc | 192 +++++++++++--------------
            be/src/exec/partitioned-aggregation-node.cc | 58 ++++----
            be/src/exec/partitioned-aggregation-node.h | 48 ++++---
            4 files changed, 148 insertions, 152 deletions

            Show
            tarmstrong Tim Armstrong added a comment - commit 0568c9ba56ecbf2d9bda28d495bb548404470f18 Author: Tim Armstrong <tarmstrong@cloudera.com> Date: Thu Nov 19 14:59:32 2015 -0800 Simplify PAGG ProcessBatch The PAGG insert logic can be simplified by exploiting the fact that HashTable::Insert will not fail unless a duplicate node is being allocated. In PAGG, duplicate nodes are never allocated. Therefore the code does not need to check for or handle these failures. As a further consequence, no special handling of the current intermediate_tuple is needed when spilling a partition. This patch also makes the code more readable by factoring the main ProcessBatch() function into several smaller functions. The main codegen'd inner loops are still inlined into ProcessBatch(). Also delete some dead code and rearrange some code to hopefully be more readable. Change-Id: I74398f067fa6398d9800eeec1db3db6fb9235160 Reviewed-on: http://gerrit.cloudera.org:8080/1509 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins be/src/exec/hash-table.h | 2 + be/src/exec/partitioned-aggregation-node-ir.cc | 192 +++++++++++-------------- be/src/exec/partitioned-aggregation-node.cc | 58 ++++---- be/src/exec/partitioned-aggregation-node.h | 48 ++++--- 4 files changed, 148 insertions , 152 deletions
            Hide
            tarmstrong Tim Armstrong added a comment -

            All subtasks are finished, so will close this parent task and start fresh for future work.

            Show
            tarmstrong Tim Armstrong added a comment - All subtasks are finished, so will close this parent task and start fresh for future work.

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development