[IMPALA-6704] session-expiry-test is flaky if no minicluster is running

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Component/s: Backend
• Epic Color: ghx-label-3
• Assignee: Philip Martin
• Reporter: Philip Martin
• Created: 19/Mar/18 20:52
• Updated: 06/Apr/18 23:24
• Resolved: 06/Apr/18 23:24

Description

I have found that on certain machines (in my case, on a 32-core machine but not on an 8-core machine), session-expiry-test is flaky if no minicluster is running. We tend to expect backend tests like this one to not require the minicluster, but in Jenkins runs, I believe the minicluster is running, hiding this flakiness.

Oddly, I found adding -alsologtostderr to the test to make it fail more reliably, on my machine. If you don't specify -alsologtostderr it mostly runs fine.

Here's the quick log:

$ be/build/debug/service/session-expiry-test --alsologtostderr
...
I0314 20:57:04.724493 15172 impala-server.cc:1237] Default query options:TQueryOptions ....
E0314 20:57:05.355170 15172 impala-server.cc:278] Could not read the root directory at hdfs://localhost:20500. Error was:
Call From XXXXXXXX to localhost:20500 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
E0314 20:57:05.355419 15172 impala-server.cc:281] Aborting Impala Server startup due to improper configuration. Impalad exiting.

I've not looked into it deeply, but naively it looks like on a slower machine, the test finishes before its impala server class has a chance to talk to HDFS.

I've attached the full log output from successful and unsuccessful runs.