Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4907

Unable to open scanner: Timed out errors when running COMPUTE STATS on Kudu-related tables

    Details

      Description

      I have noticed that when loading our test data onto a cdh5-trunk cluster, there are frequent errors when we run compute stats on Kudu-related tables, but these errors don't appear one earlier versions (e.g., with CDH 5.10). They also don't appear on mini-cluster tests.

      compute-table-stats.log from cdh5-trunk test run:

      Executing: compute stats functional_kudu.alltypestiny
        -> Error: ImpalaBeeswaxException:
       Query aborted:
      Unable to open scanner: Timed out: unable to retry before timeout: Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 465216 to be safe (mode: NON-LEADER). Current safe time: L: 390220 Physical time difference: None (Logical clock): Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 431554 to be safe (mode: NON-LEADER). Current safe time: L: 367307 Physical time difference: None (Logical clock)
      
      Executing: compute stats functional_kudu.jointbl
        -> Updated 1 partition(s) and 4 column(s).
      
      Executing: compute stats functional_kudu.emptytable
        -> Error: ImpalaBeeswaxException:
       Query aborted:
      Unable to open scanner: Timed out: Timed out waiting for ts: L: 501168 to be safe (mode: NON-LEADER). Current safe time: L: 449249 Physical time difference: None (Logical clock)
      
      Executing: compute stats functional_kudu.nulltable
        -> Error: ImpalaBeeswaxException:
       Query aborted:
      Unable to open scanner: Timed out: unable to retry before timeout: Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 535495 to be safe (mode: NON-LEADER). Current safe time: L: 438046 Physical time difference: None (Logical clock): Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 501998 to be safe (mode: NON-LEADER). Current safe time: L: 415262 Physical time difference: None (Logical clock)
      
      Executing: compute stats functional_kudu.dimtbl
        -> Error: ImpalaBeeswaxException:
       Query aborted:
      Unable to open scanner: Timed out: unable to retry before timeout: Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 570160 to be safe (mode: NON-LEADER). Current safe time: L: 461598 Physical time difference: None (Logical clock): Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 536646 to be safe (mode: NON-LEADER). Current safe time: L: 438814 Physical time difference: None (Logical clock)
      
      Executing: compute stats functional_kudu.alltypessmall
        -> Error: ImpalaBeeswaxException:
       Query aborted:
      Unable to open scanner: Timed out: unable to retry before timeout: Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 603727 to be safe (mode: NON-LEADER). Current safe time: L: 484424 Physical time difference: None (Logical clock): Remote error: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: L: 570916 to be safe (mode: NON-LEADER). Current safe time: L: 462086 Physical time difference: None (Logical clock)
      
      Executing: compute stats functional_kudu.alltypesagg_idx
        -> Updated 1 partition(s) and 15 column(s).
      
      Executing: compute stats functional_kudu.alltypesaggnonulls
        -> Updated 1 partition(s) and 14 column(s).
      
      Executing: compute stats functional_kudu.tinytable
        -> Updated 1 partition(s) and 2 column(s).
      

      compute-table-stats.log from CDH 5.10 test run:

      Executing: compute stats functional_kudu.alltypestiny
        -> Updated 1 partition(s) and 13 column(s).
      
      Executing: compute stats functional_kudu.jointbl
        -> Updated 1 partition(s) and 4 column(s).
      
      Executing: compute stats functional_kudu.emptytable
        -> Updated 1 partition(s) and 2 column(s).
      
      Executing: compute stats functional_kudu.nulltable
        -> Updated 1 partition(s) and 7 column(s).
      
      Executing: compute stats functional_kudu.dimtbl
        -> Updated 1 partition(s) and 3 column(s).
      
      Executing: compute stats functional_kudu.alltypessmall
        -> Updated 1 partition(s) and 13 column(s).
      
      Executing: compute stats functional_kudu.alltypesagg_idx
        -> Updated 1 partition(s) and 15 column(s).
      
      Executing: compute stats functional_kudu.alltypesaggnonulls
        -> Updated 1 partition(s) and 14 column(s).
      
      Executing: compute stats functional_kudu.tinytable
        -> Updated 1 partition(s) and 2 column(s).
      

        Attachments

          Activity

            People

            • Assignee:
              mjacobs Matthew Jacobs
              Reporter:
              dknupp David Knupp
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: