[IMPALA-5664] Unix time to timestamp conversions may crash impala (boost exception) - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: Impala 2.7.0
Fix Version/s: Impala 2.11.0
Component/s: Backend
Labels:
- newbie
- timestamp

Target Version:

Product Backlog
Epic Color:
ghx-label-9

Description

Unix time to timestamp conversions with dates before 1400 may crash Impala in some cases.

E.g.

[localhost:21000] > select cast(-17987443200-0.1 as timestamp);
Query: select cast(-17987443200-0.1 as timestamp)
Query submitted at: 2017-07-13 15:18:05 (Coordinator: http://mj-desktop.ca.cloudera.com:25000)
Error communicating with impalad: TSocket read 0 bytes

(gdb) bt
#0  0x00007f3a2a29dc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f3a2a2a1028 in __GI_abort () at abort.c:89
#2  0x00007f3a2adc9cbd in __gnu_cxx::__verbose_terminate_handler () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x00007f3a2adc7d46 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x00007f3a2adc7d91 in std::terminate () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5  0x00007f3a2adc7fa8 in __cxxabiv1::__cxa_throw (obj=0x964abc0, 
    tinfo=0x23825e0 <typeinfo for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_year> >>, 
    dest=0x7daed0 <boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_year> >::~clone_impl()>)
    at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_throw.cc:87
#6  0x00000000007d6596 in boost::throw_exception<boost::gregorian::bad_year> (e=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/throw_exception.hpp:69
#7  0x00000000007d1842 in boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year>::on_error ()
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:110
#8  0x00000000007cc18e in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year> >::assign (this=0x7f39d94ff7c0, value=1399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:69
#9  0x00000000007c5c4a in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year> >::constrained_value (this=0x7f39d94ff7c0, value=1399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:48
#10 0x00000000007c19aa in boost::gregorian::greg_year::greg_year (this=0x7f39d94ff7c0, year=1399)
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/greg_year.hpp:41
#11 0x00007f3a302948de in boost::date_time::gregorian_calendar_base<boost::date_time::year_month_day_base<boost::gregorian::greg_year, boost::gregorian::greg_month, boost::gregorian::greg_day>, unsigned int>::from_day_number (dayNumber=2232399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian_calendar.ipp:122
#12 0x00007f3a302985ef in boost::date_time::date<boost::gregorian::date, boost::gregorian::gregorian_calendar, boost::gregorian::date_duration>::year_month_day (
    this=0x7f39d94ff870) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/date.hpp:99
#13 0x00007f3a30298afb in boost::date_time::date_formatter<boost::gregorian::date, boost::date_time::iso_extended_format<char>, char>::date_to_string (d=...)
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/date_formatting.hpp:125
#14 0x00007f3a3029867f in boost::gregorian::to_iso_extended_string_type<char> (d=...)
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/formatters.hpp:79
#15 0x00007f3a302983d7 in boost::gregorian::to_iso_extended_string (d=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/formatters.hpp:85
#16 0x00007f3a302980d1 in impala::TimestampValue::ToString (this=0x964aae0) at /home/mj/dev/Impala/be/src/runtime/timestamp-value.cc:178
#17 0x00007f3a30297db9 in impala::operator<< (os=..., timestamp_value=...) at /home/mj/dev/Impala/be/src/runtime/timestamp-value.cc:112
#18 0x00007f3a3025feb2 in impala::RawValue::PrintValue (value=0x964aae0, type=..., scale=-1, stream=0x7f39d94ffc30) at /home/mj/dev/Impala/be/src/runtime/raw-value.cc:286
#19 0x00007f3a3025f03a in impala::RawValue::PrintValue (value=0x964aae0, type=..., scale=-1, str=0x7f39d94fffc0) at /home/mj/dev/Impala/be/src/runtime/raw-value.cc:108
#20 0x00007f3a2e849bd8 in SetTColumnValue (value=0x964aae0, type=..., col_val=0x7f39d94fffa0) at /home/mj/dev/Impala/be/src/service/fe-support.cc:143
#21 0x00007f3a2e84a73d in Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (env=0x53f09e8, caller_class=0x7f39d9500ae8, thrift_expr_batch=0x7f39d9500b00, 
    thrift_query_ctx_bytes=0x7f39d9500af8) at /home/mj/dev/Impala/be/src/service/fe-support.cc:236

The same issue may occur in BE execution as well (as opposed to FeSupport above). E.g. similarly:

[localhost:21001] > select cast(-17987443200-(tinyint_col/10.0) as timestamp) from alltypestiny where id < 2;
Query: select cast(-17987443200-(tinyint_col/10.0) as timestamp) from alltypestiny where id < 2
Query submitted at: 2017-07-13 15:27:20 (Coordinator: http://mj-desktop.ca.cloudera.com:25001)
Query progress can be monitored at: http://mj-desktop.ca.cloudera.com:25001/query_plan?query_id=554eb0b7787304cb:f6be064200000000


#0  0x00007fb90618cc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fb906190028 in __GI_abort () at abort.c:89
#2  0x00007fb906cb8cbd in __gnu_cxx::__verbose_terminate_handler () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x00007fb906cb6d46 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x00007fb906cb6d91 in std::terminate () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5  0x00007fb906cb6fa8 in __cxxabiv1::__cxa_throw (obj=0xa371dc0, 
    tinfo=0x23825e0 <typeinfo for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_year> >>, 
    dest=0x7daed0 <boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_year> >::~clone_impl()>)
    at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_throw.cc:87
#6  0x00000000007d6596 in boost::throw_exception<boost::gregorian::bad_year> (e=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/throw_exception.hpp:69
#7  0x00000000007d1842 in boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year>::on_error ()
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:110
#8  0x00000000007cc18e in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year> >::assign (this=0x7fb872b63f20, value=1399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:69
#9  0x00000000007c5c4a in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year> >::constrained_value (this=0x7fb872b63f20, value=1399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:48
#10 0x00000000007c19aa in boost::gregorian::greg_year::greg_year (this=0x7fb872b63f20, year=1399)
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/greg_year.hpp:41
#11 0x00007fb90c1838de in boost::date_time::gregorian_calendar_base<boost::date_time::year_month_day_base<boost::gregorian::greg_year, boost::gregorian::greg_month, boost::gregorian::greg_day>, unsigned int>::from_day_number (dayNumber=2232399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian_calendar.ipp:122
#12 0x00007fb90c1875ef in boost::date_time::date<boost::gregorian::date, boost::gregorian::gregorian_calendar, boost::gregorian::date_duration>::year_month_day (
    this=0x7fb872b63fd0) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/date.hpp:99
#13 0x00007fb90c187afb in boost::date_time::date_formatter<boost::gregorian::date, boost::date_time::iso_extended_format<char>, char>::date_to_string (d=...)
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/date_formatting.hpp:125
#14 0x00007fb90c18767f in boost::gregorian::to_iso_extended_string_type<char> (d=...)
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/formatters.hpp:79
#15 0x00007fb90c1873d7 in boost::gregorian::to_iso_extended_string (d=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/formatters.hpp:85
#16 0x00007fb90c1870d1 in impala::TimestampValue::ToString (this=0xa371ce0) at /home/mj/dev/Impala/be/src/runtime/timestamp-value.cc:178
#17 0x00007fb90c186db9 in impala::operator<< (os=..., timestamp_value=...) at /home/mj/dev/Impala/be/src/runtime/timestamp-value.cc:112
#18 0x00007fb90c14eeb2 in impala::RawValue::PrintValue (value=0xa371ce0, type=..., scale=-1, stream=0x7fb872b64390) at /home/mj/dev/Impala/be/src/runtime/raw-value.cc:286
#19 0x00007fb90a87905d in impala::AsciiQueryResultSet::AddOneRow (this=0x8e65bc0, col_values=..., scales=...) at /home/mj/dev/Impala/be/src/service/query-result-set.cc:172
#20 0x00007fb90b9ce97a in impala::PlanRootSink::Send (this=0x92d67e0, state=0x5870000, batch=0x8ca68f0) at /home/mj/dev/Impala/be/src/exec/plan-root-sink.cc:97
#21 0x00007fb90c10bf77 in impala::FragmentInstanceState::ExecInternal (this=0x940b480) at /home/mj/dev/Impala/be/src/runtime/fragment-instance-state.cc:276
#22 0x00007fb90c10964f in impala::FragmentInstanceState::Exec (this=0x940b480) at /home/mj/dev/Impala/be/src/runtime/fragment-instance-state.cc:89
#23 0x00007fb90c132e28 in impala::QueryState::ExecFInstance (this=0x885f800, fis=0x940b480) at /home/mj/dev/Impala/be/src/runtime/query-state.cc:321
#24 0x00007fb90c131b3a in impala::QueryState::<lambda()>::operator()(void) const (__closure=0x7fb872b64d28) at /home/mj/dev/Impala/be/src/runtime/query-state.cc:295
#25 0x00007fb90c133805 in boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#26 0x00007fb90c6085da in boost::function0<void>::operator() (this=0x7fb872b64d20)
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#27 0x00007fb90c605c67 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., 
    functor=..., thread_started=0x7fb86e35ad80) at /home/mj/dev/Impala/be/src/util/thread.cc:330
#28 0x00007fb90c60f1fc in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0xbd4f5c0, 
    f=@0xbd4f5b8: 0x7fb90c605948 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
    at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#29 0x00007fb90c60f13f in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (
    this=0xbd4f5b8) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#30 0x00007fb90c60f102 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0xbd4f400) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#31 0x000000000087d35a in thread_proxy ()
#32 0x00007fb906527184 in start_thread (arg=0x7fb872b65700) at pthread_create.c:312
#33 0x00007fb906253ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

It looks like the issue is that code that uses boost time_duration addition/subtraction may create a ptime with an invalid date, yet the ptime/date does not indicate that it is is_special() that we normally check (e.g. in TimestampValue::HasDate().

The code in timestamp-functions-ir.cc AddSub has an obscure call to ptime::date()::year() which isn't explained, but looking at the git history there is a relevant comment:

From commit cf60967b:

-/// The AddInterval() functions provide a unified interface for adding intervals of all
-/// types. To subtract, the 'interval' can be negative. The default template below only
-/// handles time intervals (hours, seconds, etc). Date intervals (days, weeks, etc) need
-/// to be specified individually.
-template <typename Interval>
-inline TimestampVal AddInterval(const TimestampValue& timestamp, int64_t interval) {
-  ptime temp;
-  timestamp.ToPtime(&temp);
-  temp += Interval(interval);
-  temp.date().year();  // Forces validation on the year (which will throw).
-  const TimestampValue value(temp);
-  TimestampVal val;
-  value.ToTimestampVal(&val);
-  return val;

A bit more digging shows this related comment:

  // Adding/subtracting boost::gregorian::dates can throw if the result exceeds the
  // min/max supported dates. (Sometimes the exception is thrown lazily and calling an
  // accessor functions is needed to trigger validation.)

It seems we need additional validation after using date addition/subtraction. It seems like calling year() w/in a try/catch block seems to work and we can start with that as a solution. Maybe there's a better way to do this, worth exploring in the future but I don't think has to hold up a fix for this.

It looks like the FromUnixTime functions that take subseconds, microseconds, and nanoseconds may all be susceptible. The patch should include test cases to cover all of these fns.

Attachments

Issue Links

is depended upon by

IMPALA-5668 Subsecond Unix times around the first supported TIMESTAMP may be wrong

Resolved

is related to

IMPALA-3598 Timestamp casting is inconsistent

Resolved

Unix time to timestamp conversions may crash impala (boost exception)

Details

Description

Attachments

Issue Links

Activity

People

Dates