[IMPALA-10058] Kudu queries hit error "Unable to deserialize scan token"

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: broken-build flaky
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-5
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 06/Aug/20 22:37
• Updated: 17/Nov/20 21:03
• Resolved: 04/Nov/20 16:02

Description

I have seen a few test runs fail with a large number of Kudu tests failing with:

ImpalaBeeswaxException: ImpalaBeeswaxException:  Query aborted:Unable to deserialize scan token for node with id '0' for Kudu table 'impala::functional_kudu.alltypestiny': Not found: the table does not exist: table_name: ""

In the Impalad log, the errors looks like this:

I0804 18:35:19.075631 18788 status.cc:129] 1c4fdf1a9de8d577:32d91ed500000002] Unable to deserialize scan token for node with id '1' for Kudu table 'impala::functional_kudu.alltypessmall': Not found: the table does not exist: table_name: ""
    @          0x1cabde1  impala::Status::Status()
    @          0x28e97a9  impala::KuduScanner::OpenNextScanToken()
    @          0x284eb33  impala::KuduScanNode::ProcessScanToken()
    @          0x284f15e  impala::KuduScanNode::RunScannerThread()
    @          0x284e351  _ZZN6impala12KuduScanNode17ThreadAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
    @          0x284fa48  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12KuduScanNode17ThreadAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
    @          0x2062c1d  boost::function0<>::operator()()
    @          0x26867cf  impala::Thread::SuperviseThread()
    @          0x268e76c  boost::_bi::list5<>::operator()<>()
    @          0x268e690  boost::_bi::bind_t<>::operator()()
    @          0x268e651  boost::detail::thread_data<>::run()
    @          0x3e616a1  thread_proxy
    @     0x7fe872ef4e24  start_thread
    @     0x7fe86f9c934c  __clone

This error would be coming from the Kudu client in this code in KuduScanner:

  kudu::client::KuduScanner* scanner;
  KUDU_RETURN_IF_ERROR(kudu::client::KuduScanToken::DeserializeIntoScanner(
                           scan_node_->kudu_client(), scan_token, &scanner),
      BuildErrorString("Unable to deserialize scan token"));

This has happened multiple times in the docker-based tests, but I have also seen a couple jobs with the normal test runs.

Issue Links

• is related to: IMPALA-8751 Kudu tables cannot be found after created