[IMPALA-3669] several test_scratch_disk failures when run against S3 due to different memory requirements

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.6.0
• Component/s: Backend
• Labels: broken-build
• Target Version: Impala 2.6.0
• Assignee: Tim Armstrong
• Reporter: Michael Brown
• Created: 03/Jun/16 18:47
• Updated: 07/Jun/16 16:07
• Resolved: 07/Jun/16 16:07

Description

The test_scrach_disk tests are currently skipped when run against S3 in Jenkins. When we fix that with IMPALA-3614 (review https://gerrit.cloudera.org/#/c/3265/ ), the tests now run, but they fail.

http://sandbox.jenkins.cloudera.com/view/Impala/view/Workflow/job/impala-umbrella-build-and-test-s3/54/

The failure comes from the fact that the query expected to spill (and fail) ends up succeeding.

It seems that on S3, the expected-to-spill query doesn't need 200M, the mem limit we set to make it spill.

Should such a discrepancy exist?

I collected profiles of both the "expected to spill" and "expected not to spill" queries used in the test, both for HDFS and S3. I'll attach them here.

---

It's possible to run tests locally against S3. You need a S3 bucket and keys. It worked for me to set environment variables accordingly, re-source impala-config.sh, and proceed:

$ cat ~/.awsrc-REDACTED 
export S3_BUCKET="your bucket name"

export AWS_ACCESS_KEY_ID="your aws access key"
export AWS_SECRET_ACCESS_KEY="your aws secret key"

export TARGET_FILESYSTEM="s3"
export FILESYSTEM_PREFIX="s3a://${S3_BUCKET}"
$ source ~/.awsrc-REDACTED 
$ source bin/impala-config.sh
$ ./buildall.sh

There is a bucket already loaded with data I think you can use without having to reformat. sailesh might be able to help if you get stuck on this part if I can't help.