Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4554

Memory corruption of nested collection with MT_DOP > 0.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:

      Description

      A query that explodes and aggregates a nested collection appears to hang with mt_dop>0.

      The reason is that the the in-memory collection slot has a "garbage" size that leads to infinite unnesting. It looks like the memory gets corrupted at some point, but I have not been able to make out where exactly yet.

      Start Impala like this:

      bin/start-impala-cluster.py -s 1 --impalad_args="--default_query_options=mt_dop=1"
      

      Query to repro:

      select id, cnt from functional_parquet.complextypestbl t,
      (select count(item) cnt from t.int_array) v;
      

      Relevant code snippet from unnest-node.cc

        Tuple* tuple = containing_subplan_->current_input_row_->GetTuple(coll_tuple_idx_);
        if (tuple != NULL) {
          // Retrieve the collection value to be unnested directly from the tuple. We purposely
          // ignore the null bit of the slot because we may have set it in a previous Open() of
          // this same unnest node for projection.
          coll_value_ = reinterpret_cast<const CollectionValue*>(
              tuple->GetSlot(coll_slot_desc_->tuple_offset()));
          // Projection: Set the slot containing the collection value to NULL.
          tuple->SetNull(coll_slot_desc_->null_indicator_offset());
        } else {
          coll_value_ = &EMPTY_COLLECTION_VALUE;
          DCHECK_EQ(coll_value_->num_tuples, 0);
        }
      
      At this point coll_value_->num_tuples appears to be garbage.
      

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              alex.behm Alexander Behm
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: