Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1619

Support single allocations larger 1GB

    XMLWordPrintableJSON

    Details

      Description

      Currently, most of the allocations are managed via the FreePool class. Unfortunately, this class has a sever drawback that it uses a signed 32 bit integer to specify allocations. Interestingly, the FreePool manages a list of 64 sub-lists with free blocks, but the upper half of these lists is never used. The FreePool uses a MemPool to perform the actual allocation, but the MemPool only uses a 32 bit signed integer as well to indicate the allocation size.

      If an allocation requests more than 1GB of data, the 32 bit integer will overflow and request a negative amount of memory which will ultimately lead to a crash.

      To reproduce the crash issue:

      select length(group_concat(l_comment, "!")) from (select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem union all
      select l_comment from tpch_parquet.lineitem) a;
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kwho Michael Ho
                Reporter:
                mgrund_impala_bb91 Martin Grund
              • Votes:
                1 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: