Details

      Description

      After IMPALA-2736 there was an increase in peak memory consumption mostly due to the Parquet scanner.
      In most cases the Parquet scanner ends up buffering more batches than needed.

      In the attached profile the scanner memory increases from 2.17GB to 3.3GB.

      Workarounds
      The following query options may help to reduce scanner memory consumption:

      • Reduce the number of scanner threads (set num_scanner_threads=30)
      • Reduce the batch size (set batch_size=512)

      Of course, increasing the mem limit may also help.

      1. tpch_q14_2.5.txt
        278 kB
        Mostafa Mokhtar
      2. tpch_q14_2.6.txt
        281 kB
        Mostafa Mokhtar

        Activity

        Hide
        jrussell John Russell added a comment -

        Added to Known Issues. Blanked out "doc text" field so this issue doesn't show up on my to-do list.

        Show
        jrussell John Russell added a comment - Added to Known Issues. Blanked out "doc text" field so this issue doesn't show up on my to-do list.
        Hide
        kwho Michael Ho added a comment -

        The increase in memory usage is probably due to the removal of the following check
        in HdfsParquetScanner::AssembleRows() which limits the amount of memory consumed by
        a row batch. This is a regression against Impala 2.5 but not necessarily against 2.3.

                // Exit this loop early if the batch exceeds memory usage capacity
                if (pool->total_allocated_bytes() >= RowBatch::AT_CAPACITY_MEM_USAGE) {
                  // Update num_rows read and break
                  num_rows = i + 1;
                  break;
                }
        
        Show
        kwho Michael Ho added a comment - The increase in memory usage is probably due to the removal of the following check in HdfsParquetScanner::AssembleRows() which limits the amount of memory consumed by a row batch. This is a regression against Impala 2.5 but not necessarily against 2.3. // Exit this loop early if the batch exceeds memory usage capacity if (pool->total_allocated_bytes() >= RowBatch::AT_CAPACITY_MEM_USAGE) { // Update num_rows read and break num_rows = i + 1; break; }
        Hide
        HuaisiXu Huaisi Xu added a comment -

        Talked with Michael, we think that this is caused by IMPALA-2473, and we have reverted part of that in 5.7.3. So this is not a regression from >5.7.3. Remove the target version for now. Feel free to re-tag it once this is about to be fixed.

        Show
        HuaisiXu Huaisi Xu added a comment - Talked with Michael, we think that this is caused by IMPALA-2473 , and we have reverted part of that in 5.7.3. So this is not a regression from >5.7.3. Remove the target version for now. Feel free to re-tag it once this is about to be fixed.
        Hide
        kwho Michael Ho added a comment -

        Memory usage should decrease with this commit: https://git-wip-us.apache.org/repos/asf?p=incubator-impala.git;a=commit;h=1522da3510a36635e3fc694b26211554fcd2793a
        It won't go back to the previous level due to the revert of IMPALA-2473.

        IMPALA-3662: Don't double allocate tuples buffer in parquet scanner

        HdfsScanner::StartNewRowBatch() is called once per row batch
        by the parquet scanner to allocate a new row batch and tuple
        buffer. Similarly, a scratch batch is created for each row
        batch in HdfsParquetScanner::AssembleRows() which also contains
        the tuple buffer. In reality, only the tuple buffer in the
        scratch batch is used. So, the tuple buffer allocated by
        HdfsScanner::StartNewRowBatch() is unused memory for the
        parquet scanner.

        This change fixes the problem above by implementing
        HdfsParquetScanner::StartNewRowBatch() which creates
        a new row batch without allocating the tuple buffer.
        With this patch, the memory consumption when
        materializing very wide tuples is reduced by half.

        Change-Id: I826061a2be10fd0528ca4dd1e97146e3cb983370
        Reviewed-on: http://gerrit.cloudera.org:8080/4064
        Reviewed-by: Michael Ho <kwho@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        kwho Michael Ho added a comment - Memory usage should decrease with this commit: https://git-wip-us.apache.org/repos/asf?p=incubator-impala.git;a=commit;h=1522da3510a36635e3fc694b26211554fcd2793a It won't go back to the previous level due to the revert of IMPALA-2473 . IMPALA-3662 : Don't double allocate tuples buffer in parquet scanner HdfsScanner::StartNewRowBatch() is called once per row batch by the parquet scanner to allocate a new row batch and tuple buffer. Similarly, a scratch batch is created for each row batch in HdfsParquetScanner::AssembleRows() which also contains the tuple buffer. In reality, only the tuple buffer in the scratch batch is used. So, the tuple buffer allocated by HdfsScanner::StartNewRowBatch() is unused memory for the parquet scanner. This change fixes the problem above by implementing HdfsParquetScanner::StartNewRowBatch() which creates a new row batch without allocating the tuple buffer. With this patch, the memory consumption when materializing very wide tuples is reduced by half. Change-Id: I826061a2be10fd0528ca4dd1e97146e3cb983370 Reviewed-on: http://gerrit.cloudera.org:8080/4064 Reviewed-by: Michael Ho <kwho@cloudera.com> Tested-by: Internal Jenkins
        Hide
        srus Silvius Rus added a comment -

        Michael Ho, Dan Hecht, is this fixed?

        Show
        srus Silvius Rus added a comment - Michael Ho , Dan Hecht , is this fixed?
        Hide
        kwho Michael Ho added a comment -

        As mentioned in the previous update, the increase in memory was due to the partial un-doing of IMPALA-2473 which limits the size of varlen data in a row batch. As it turns out, limiting the size of varlen data in a row batch will lead to non-trivial regression for very wide row (e.g. CDH-41243). The commit https://github.com/apache/incubator-impala/commit/1522da3510a36635e3fc694b26211554fcd2793a already reduces some of the memory usage introduced by IMPALA-2736.

        Show
        kwho Michael Ho added a comment - As mentioned in the previous update, the increase in memory was due to the partial un-doing of IMPALA-2473 which limits the size of varlen data in a row batch. As it turns out, limiting the size of varlen data in a row batch will lead to non-trivial regression for very wide row (e.g. CDH-41243). The commit https://github.com/apache/incubator-impala/commit/1522da3510a36635e3fc694b26211554fcd2793a already reduces some of the memory usage introduced by IMPALA-2736 .

          People

          • Assignee:
            kwho Michael Ho
            Reporter:
            mmokhtar Mostafa Mokhtar
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development