Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 2.9.0
Description
I have noticed that when loading our test data onto a cdh5-trunk cluster, there are frequent errors when we run compute stats on Kudu-related tables, but these errors don't appear one earlier versions (e.g., with CDH 5.10). They also don't appear on mini-cluster tests.
compute-table-stats.log from cdh5-trunk test run:
Executing: compute stats functional_kudu.alltypestiny -> Error: ImpalaBeeswaxException: Query aborted: Unable to open scanner: Timed out: unable to retry before timeout: Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 465216 to be safe (mode: NON-LEADER). Current safe time: L: 390220 Physical time difference: None (Logical clock): Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 431554 to be safe (mode: NON-LEADER). Current safe time: L: 367307 Physical time difference: None (Logical clock) Executing: compute stats functional_kudu.jointbl -> Updated 1 partition(s) and 4 column(s). Executing: compute stats functional_kudu.emptytable -> Error: ImpalaBeeswaxException: Query aborted: Unable to open scanner: Timed out: Timed out waiting for ts: L: 501168 to be safe (mode: NON-LEADER). Current safe time: L: 449249 Physical time difference: None (Logical clock) Executing: compute stats functional_kudu.nulltable -> Error: ImpalaBeeswaxException: Query aborted: Unable to open scanner: Timed out: unable to retry before timeout: Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 535495 to be safe (mode: NON-LEADER). Current safe time: L: 438046 Physical time difference: None (Logical clock): Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 501998 to be safe (mode: NON-LEADER). Current safe time: L: 415262 Physical time difference: None (Logical clock) Executing: compute stats functional_kudu.dimtbl -> Error: ImpalaBeeswaxException: Query aborted: Unable to open scanner: Timed out: unable to retry before timeout: Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 570160 to be safe (mode: NON-LEADER). Current safe time: L: 461598 Physical time difference: None (Logical clock): Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 536646 to be safe (mode: NON-LEADER). Current safe time: L: 438814 Physical time difference: None (Logical clock) Executing: compute stats functional_kudu.alltypessmall -> Error: ImpalaBeeswaxException: Query aborted: Unable to open scanner: Timed out: unable to retry before timeout: Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 603727 to be safe (mode: NON-LEADER). Current safe time: L: 484424 Physical time difference: None (Logical clock): Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 570916 to be safe (mode: NON-LEADER). Current safe time: L: 462086 Physical time difference: None (Logical clock) Executing: compute stats functional_kudu.alltypesagg_idx -> Updated 1 partition(s) and 15 column(s). Executing: compute stats functional_kudu.alltypesaggnonulls -> Updated 1 partition(s) and 14 column(s). Executing: compute stats functional_kudu.tinytable -> Updated 1 partition(s) and 2 column(s).
compute-table-stats.log from CDH 5.10 test run:
Executing: compute stats functional_kudu.alltypestiny -> Updated 1 partition(s) and 13 column(s). Executing: compute stats functional_kudu.jointbl -> Updated 1 partition(s) and 4 column(s). Executing: compute stats functional_kudu.emptytable -> Updated 1 partition(s) and 2 column(s). Executing: compute stats functional_kudu.nulltable -> Updated 1 partition(s) and 7 column(s). Executing: compute stats functional_kudu.dimtbl -> Updated 1 partition(s) and 3 column(s). Executing: compute stats functional_kudu.alltypessmall -> Updated 1 partition(s) and 13 column(s). Executing: compute stats functional_kudu.alltypesagg_idx -> Updated 1 partition(s) and 15 column(s). Executing: compute stats functional_kudu.alltypesaggnonulls -> Updated 1 partition(s) and 14 column(s). Executing: compute stats functional_kudu.tinytable -> Updated 1 partition(s) and 2 column(s).