Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
  3. IMPALA-3205

Validate and fix spilling performance and memory usage of new buffer pool

    Details

      Description

      We want to ensure that performance doesn't regress:

      • Memory usage for the new buffer pool (with the old small buffers optimisation removed) should be similar or less than the old code for typical workloads. This will require reducing the buffer size - e.g. to 1MB or less.
      • Spill-to-disk performance should not regress more than is acceptable. This may be tricky to get right since the I/O size will be reduced from 8MB. We may need to allocate logically sequential blocks in contiguous disk and do some kind of prefetching in the buffer pool to issue multiple writes at a time.

        Attachments

        1. spilling-sort-bufferpool.txt
          4 kB
          Tim Armstrong
        2. spilling-sort-bbm.txt
          2 kB
          Tim Armstrong
        3. spilling-hj2-bp-8mb-buffers.txt
          33 kB
          Tim Armstrong
        4. spilling-hj2-bp.txt
          56 kB
          Tim Armstrong
        5. spilling-hj2-bbm.txt
          98 kB
          Tim Armstrong
        6. spilling-agg-bufferpool-8mb-buffers.txt
          55 kB
          Tim Armstrong
        7. spilling-agg-bufferpool.txt
          55 kB
          Tim Armstrong
        8. spilling-agg-bbm.txt
          27 kB
          Tim Armstrong
        9. min_mem.py
          0.5 kB
          Tim Armstrong
        10. min_mem_tpcds_5_bp.json
          55 kB
          Tim Armstrong
        11. min_mem_tpcds_5_bp.csv
          0.3 kB
          Tim Armstrong
        12. min_mem_tpcds_5_bp_reduce_min_remaining.json
          55 kB
          Tim Armstrong
        13. min_mem_tpcds_5_bp_reduce_min_remaining.csv
          0.3 kB
          Tim Armstrong
        14. min_mem_tpcds_5_bbm.json
          55 kB
          Tim Armstrong
        15. min_mem_tpcds_5_bbm.csv
          0.3 kB
          Tim Armstrong
        16. min_mem_part_aggs_joins.csv
          0.2 kB
          Tim Armstrong
        17. min_mem_old_aggs_joins.csv
          0.2 kB
          Tim Armstrong
        18. min_mem_buffer_pool_default.csv
          0.2 kB
          Tim Armstrong
        19. min_mem_buffer_pool_64kb_buffers.csv
          0.2 kB
          Tim Armstrong

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: