[IMPALA-7426] T-test is an unreliable method for comparing non-normal distributions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.1.0
• Component/s: Infrastructure
• Epic Color: ghx-label-3
• Assignee: Jim Apple
• Reporter: Jim Apple
• Created: 11/Aug/18 17:38
• Updated: 12/Sep/18 21:16
• Resolved: 12/Sep/18 21:16

Description

https://en.wikipedia.org/wiki/Student%27s_t-test is for normally distributed variables, and many Impala benchmarks won't be formally distributed. In particular, none of them will have negative run times.

We should consider https://en.wikipedia.org/wiki/Nonparametric_statistics like https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test#Comparison_to_Student's_t-test in report_benchmark_results.py.