Details

      Description

      Given that unaligned memory accesses have been getting faster on modern architectures, we should revisit our tuple memory layout which adds padding to avoid unaligned accesses.

      The code for computing our mem layout is in TupldDescriptor.java, and changes in the layout need to be reflected in descriptors.cc TupleDescriptor::GenerateLlvmStruct().

      I did a simple experiment (diff attached), where we switch to a packed layout, and the results look encouraging.

      Perf results vs. cdh5-trunk are here:

      TPCH-300
      http://sandbox.jenkins.cloudera.com/view/Impala/view/Cluster%20Runs/view/10-node-cdh5/job/impala-workload-runner-10node-cdh5/1239/

      TPCDS-500
      http://sandbox.jenkins.cloudera.com/view/Impala/view/Cluster%20Runs/view/10-node-cdh5/job/impala-workload-runner-10node-cdh5/1238/

      I think we could further optimize the layout by organizing the slots in descending order of their size, and by putting the null bits last. We could also pack var-len slots into 12 bytes (4 byte len + ptr) instead of 16 (4 byte len padded + ptr)

        Activity

        Show
        alex.behm Alexander Behm added a comment - Related: https://issues.cloudera.org/browse/IMPALA-2680
        Hide
        tarmstrong Tim Armstrong added a comment -

        This looks like an easy win.

        Show
        tarmstrong Tim Armstrong added a comment - This looks like an easy win.
        Hide
        mjacobs Matthew Jacobs added a comment -

        If we use Kudu's mem layout then we get a big win not having to convert row batches we get from Kudu. (added tag 'kudu')

        Show
        mjacobs Matthew Jacobs added a comment - If we use Kudu's mem layout then we get a big win not having to convert row batches we get from Kudu. (added tag 'kudu')
        Hide
        alex.behm Alexander Behm added a comment -

        Agreed, we should just adopt Kudu's format.

        Show
        alex.behm Alexander Behm added a comment - Agreed, we should just adopt Kudu's format.
        Hide
        alex.behm Alexander Behm added a comment -

        Increasing priority to reflect that this change will help unlock perf improvements for Kudu scans.

        Show
        alex.behm Alexander Behm added a comment - Increasing priority to reflect that this change will help unlock perf improvements for Kudu scans.
        Hide
        alex.behm Alexander Behm added a comment -

        commit b0e87c685d96e4e55a0fda3b23c10fc069f7551a
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Tue Dec 22 13:56:32 2015 -0800

        IMPALA-2789: More compact mem layout with null bits at the end.

        There are two motivations for this change:
        1. Reduce memory consumption.
        2. Pave the way for full memory layout compatibility between
        Impala and Kudu to eventually enable zero-copy scans. This
        patch is a only first step towards that goal.

        New Memory Layout
        Slots are placed in descending order by size with trailing bytes to
        store null flags. Null flags are omitted for non-nullable slots. There
        is no padding between tuples when stored back-to-back in a row batch.

        Example: select bool_col, int_col, string_col, smallint_col
        from functional.alltypes
        Slots: string_col|int_col|smallint_col|bool_col|null_byte
        Offsets: 0 16 20 22 23

        The main change is to move the null indicators to the end of tuples.
        The new memory layout is fully packed with no padding in between
        slots or tuples.

        Performance:
        Our standard cluster perf tests showed no significant difference in
        query response times as well as consumed cycles, and a slight
        reduction in peak memory consumption.

        Testing:
        An exhaustive test run passed. Ran a few select tests like TPC-H/DS
        with ASAN locally.

        These follow-on changes are planned:
        1. Planner needs to mark slots non-nullable if they correspond
        to a non-nullable Kudu column.
        2. Update Kudu scan node to copy tuples with memcpy.
        3. Kudu client needs to support transferring ownership of the
        tuple memory (maybe do direct and indirect buffers separately).
        4. Update Kudu scan node to use memory transfer instead of copy

        Change-Id: Ib6510c75d841bddafa6638f1bd2ac6731a7053f6
        Reviewed-on: http://gerrit.cloudera.org:8080/4673
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        alex.behm Alexander Behm added a comment - commit b0e87c685d96e4e55a0fda3b23c10fc069f7551a Author: Alex Behm <alex.behm@cloudera.com> Date: Tue Dec 22 13:56:32 2015 -0800 IMPALA-2789 : More compact mem layout with null bits at the end. There are two motivations for this change: 1. Reduce memory consumption. 2. Pave the way for full memory layout compatibility between Impala and Kudu to eventually enable zero-copy scans. This patch is a only first step towards that goal. New Memory Layout Slots are placed in descending order by size with trailing bytes to store null flags. Null flags are omitted for non-nullable slots. There is no padding between tuples when stored back-to-back in a row batch. Example: select bool_col, int_col, string_col, smallint_col from functional.alltypes Slots: string_col|int_col|smallint_col|bool_col|null_byte Offsets: 0 16 20 22 23 The main change is to move the null indicators to the end of tuples. The new memory layout is fully packed with no padding in between slots or tuples. Performance: Our standard cluster perf tests showed no significant difference in query response times as well as consumed cycles, and a slight reduction in peak memory consumption. Testing: An exhaustive test run passed. Ran a few select tests like TPC-H/DS with ASAN locally. These follow-on changes are planned: 1. Planner needs to mark slots non-nullable if they correspond to a non-nullable Kudu column. 2. Update Kudu scan node to copy tuples with memcpy. 3. Kudu client needs to support transferring ownership of the tuple memory (maybe do direct and indirect buffers separately). 4. Update Kudu scan node to use memory transfer instead of copy Change-Id: Ib6510c75d841bddafa6638f1bd2ac6731a7053f6 Reviewed-on: http://gerrit.cloudera.org:8080/4673 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development