Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.11.0
    • Impala 2.11.0
    • Backend
    • ghx-label-4

    Description

      A DCHECK was hit in an ASAN build when running query_test/test_udfs.py::TestUdfExecution::test_java_udfs

      From impalad.FATAL:

      F1113 02:38:42.816524 23431 scalar-expr-evaluator.cc:70] Check failed: !initialized_ || closed_ 
      F1113 02:38:42.878442 23486 scalar-expr-evaluator.cc:70] Check failed: !initialized_ || closed_ 
      F1113 02:38:42.878442 23486 scalar-expr-evaluator.cc:70] Check failed: !initialized_ || closed_ F1113 02:38:42.898176 23490 scalar-expr-evaluator.cc:70] Check failed: !initialized_ || closed_ 
      

      Attachments

        Issue Links

        Activity

          Wasn't really sure who to assign this to, so I chose you, Michael. This started happening recently and I couldn't find a relevant commit that could cause this. Maybe this somehow has to do with your recent patch?

          tarasbob Taras Bobrovytsky added a comment - Wasn't really sure who to assign this to, so I chose you, Michael. This started happening recently and I couldn't find a relevant commit that could cause this. Maybe this somehow has to do with your recent patch?
          tarmstrong Tim Armstrong added a comment -

          Pretty sure this is a genuine bug in ScalarExprEvaluator where it isn't cleaned up on some error path. We've seen a lot of failures with random errors related to UDFs. My suspicion is that IMPALA-6001 (function metadata errors) somehow triggers these errors non-deterministically.

          tarmstrong Tim Armstrong added a comment - Pretty sure this is a genuine bug in ScalarExprEvaluator where it isn't cleaned up on some error path. We've seen a lot of failures with random errors related to UDFs. My suspicion is that IMPALA-6001 (function metadata errors) somehow triggers these errors non-deterministically.
          kwho Michael Ho added a comment - - edited

          Yes, the bug is here:

          Status ScalarExprEvaluator::Clone(ObjectPool* pool, RuntimeState* state,
              MemPool* expr_perm_pool, MemPool* expr_results_pool,
              const vector<ScalarExprEvaluator*>& evals,
              vector<ScalarExprEvaluator*>* cloned_evals) {
            DCHECK(cloned_evals != nullptr);
            DCHECK(cloned_evals->empty());
            for (int i = 0; i < evals.size(); ++i) {
              ScalarExprEvaluator* cloned_eval;
              RETURN_IF_ERROR(
                  evals[i]->Clone(pool, state, expr_perm_pool, expr_results_pool, &cloned_eval)); <<----
              cloned_evals->push_back(cloned_eval);
            }
            return Status::OK();
          }
          
          kwho Michael Ho added a comment - - edited Yes, the bug is here: Status ScalarExprEvaluator::Clone(ObjectPool* pool, RuntimeState* state, MemPool* expr_perm_pool, MemPool* expr_results_pool, const vector<ScalarExprEvaluator*>& evals, vector<ScalarExprEvaluator*>* cloned_evals) { DCHECK(cloned_evals != nullptr); DCHECK(cloned_evals->empty()); for (int i = 0; i < evals.size(); ++i) { ScalarExprEvaluator* cloned_eval; RETURN_IF_ERROR( evals[i]->Clone(pool, state, expr_perm_pool, expr_results_pool, &cloned_eval)); <<---- cloned_evals->push_back(cloned_eval); } return Status::OK(); }
          kwho Michael Ho added a comment -

          Fixed at https://github.com/apache/incubator-impala/commit/3ddafcd29505614a01c8f4362396635c84ab4052

          IMPALA-6184: Clean up after ScalarExprEvaluator::Clone() fails
          When ScalarExprEvaluator::Clone() fails, the newly created evaluator was
          not added to the output vector. This makes it impossible for callers to
          close and clean up the evaluators afterwards. This change fixes this by
          always adding the newly created evaluator to the output vector before
          checking for the error status.

          This path is only exercised in the scanner code. Two new tests are added
          to exercise the failure paths.

          Testing done: newly added tests in udf-errors.test

          Change-Id: I45ffd722d0a69ad05ae3c748cf504c7f1a959a1d
          Reviewed-on: http://gerrit.cloudera.org:8080/8572
          Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
          Tested-by: Impala Public Jenkins

          kwho Michael Ho added a comment - Fixed at https://github.com/apache/incubator-impala/commit/3ddafcd29505614a01c8f4362396635c84ab4052 IMPALA-6184 : Clean up after ScalarExprEvaluator::Clone() fails When ScalarExprEvaluator::Clone() fails, the newly created evaluator was not added to the output vector. This makes it impossible for callers to close and clean up the evaluators afterwards. This change fixes this by always adding the newly created evaluator to the output vector before checking for the error status. This path is only exercised in the scanner code. Two new tests are added to exercise the failure paths. Testing done: newly added tests in udf-errors.test Change-Id: I45ffd722d0a69ad05ae3c748cf504c7f1a959a1d Reviewed-on: http://gerrit.cloudera.org:8080/8572 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins

          People

            kwho Michael Ho
            tarasbob Taras Bobrovytsky
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment