[IMPALA-10691] Impala crashes sporadically when multiple CAST(FORMAT)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: crash
• Epic Color: ghx-label-12
• Assignee: Csaba Ringhofer
• Reporter: Gabor Kaszab
• Created: 30/Apr/21 11:56
• Updated: 04/May/21 04:50
• Resolved: 04/May/21 04:50

Description

One way to repro. This crashes Impala after a few runs.

use functional;
set max_scan_range_length=16384;
set mt_dop=2;
select count(*) from alltypes a1, alltypes a2 where cast(cast(a1.date_string_col as date format 'MM/dd/yy') as string format 'MM/dd/yy') = cast(cast(a2.date_string_col as date format 'MM/dd/yy') as string format 'MM/dd/yy');

The trace can be different but this is one frequently coming up:

F0430 12:32:43.029990  6427 datetime-iso-sql-format-tokenizer.cc:138] 614cc7aa439af2d5:96e9c4c200000002] Check failed: str_begin <= *current_pos && *current_pos < str_end
*** Check failure stack trace: ***
    @          0x548af0c  google::LogMessage::Fail()
    @          0x548c7fc  google::LogMessage::SendToLog()
    @          0x548a86a  google::LogMessage::Flush()
    @          0x548e468  google::LogMessageFatal::~LogMessageFatal()
    @          0x24d619c  impala::datetime_parse_util::IsoSqlFormatTokenizer::ProcessNextToken()
    @          0x24d5d9e  impala::datetime_parse_util::IsoSqlFormatTokenizer::Tokenize()
    @          0x2cce85c  impala::CastFormatExpr::OpenEvaluator()
    @          0x2c4e43f  impala::ScalarExpr::OpenEvaluator()
    @          0x2c93690  impala::ScalarFnCall::OpenEvaluator()
    @          0x2cce5b9  impala::CastFormatExpr::OpenEvaluator()
    @          0x2c55959  impala::ScalarExprEvaluator::Open()
    @          0x2c559e8  impala::ScalarExprEvaluator::Open()
    @          0x2b9acb8  impala::HashTableCtx::Open()
    @          0x2ad8725  impala::PartitionedHashJoinNode::Open()
    @          0x2bc1c86  impala::AggregationNode::Open()
    @          0x24a61df  impala::FragmentInstanceState::Open()
    @          0x24a27ea  impala::FragmentInstanceState::Exec()
    @          0x23e65af  impala::QueryState::ExecFInstance()

Another possible trace is:

#7  0x000000000239d0a1 in impala::DateParser::Format[abi:cxx11](impala::datetime_parse_util::DateTimeFormatContext const&, impala::DateValue const&) (dt_ctx=..., date=...)
    at /home/boroknagyz/Impala/be/src/runtime/date-parse-util.cc:135
#8  0x000000000227a5d7 in impala::DateValue::Format[abi:cxx11](impala::datetime_parse_util::DateTimeFormatContext const&) const (this=0x7f21674d268c, dt_ctx=...)
    at /home/boroknagyz/Impala/be/src/runtime/date-value.cc:152
#9  0x0000000002cccdd2 in impala::CastFunctions::CastToStringVal (ctx=0x93bcee8, val=...) at /home/boroknagyz/Impala/be/src/exprs/cast-functions-ir.cc:219
#10 0x0000000002cab261 in impala::ScalarFnCall::InterpretEval<impala_udf::StringVal> (this=0xcf29c00, eval=0xaa8a1a0, row=0xd08a000) at /home/boroknagyz/Impala/be/src/exprs/scalar-fn-call.cc:490
#11 0x0000000002c90a7d in impala::ScalarFnCall::GetStringValInterpreted (this=0xcf29c00, eval=0xaa8a1a0, row=0xd08a000) at /home/boroknagyz/Impala/be/src/exprs/scalar-fn-call.cc:556
#12 0x0000000002c5c330 in impala::ScalarExpr::GetStringVal (this=0xcf29c00, eval=0xaa8a1a0, row=0xd08a000) at /home/boroknagyz/Impala/be/src/exprs/scalar-expr.inline.h:62
#13 0x0000000002c595a7 in impala::ScalarExprEvaluator::GetValue (this=0xaa8a1a0, expr=..., row=0xd08a000) at /home/boroknagyz/Impala/be/src/exprs/scalar-expr-evaluator.cc:304
#14 0x0000000002c59281 in impala::ScalarExprEvaluator::GetValue (this=0xaa8a1a0, row=0xd08a000) at /home/boroknagyz/Impala/be/src/exprs/scalar-expr-evaluator.cc:255
#15 0x00000000023d9798 in impala::KrpcDataStreamSender::HashRow (this=0x1363f200, row=0xd08a000) at /home/boroknagyz/Impala/be/src/runtime/krpc-data-stream-sender.cc:977
#16 0x00000000023e3cfc in impala::KrpcDataStreamSender::HashAndAddRows (this=0x1363f200, batch=0xaa8a0d0) at /home/boroknagyz/Impala/be/src/runtime/krpc-data-stream-sender-ir.cc:38

There is another scenario that gives an error incorrectly:

[localhost:21050] functional> select count(*) from alltypes a1, alltypes a2 where cast(cast(a1.date_string_col as date format 'MM/dd/yy') as string format 'yyyyMMdd') = cast(cast(a2.date_string_col as date format 'MM/dd/yy') as string format 'yyyyMMdd');
...
ERROR: UDF ERROR: String to Date parse failed. Input '07/03/10' doesn't match with format 'yyyyMMdd'

A TSAN build shows data race pointing here:

https://github.com/apache/impala/blob/bca1b43efb72d839962f9e6999ca747fca7f17d9/be/src/exprs/cast-format-expr.cc#L37