Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.8.0
-
None
Description
I brought up this idea on the mailing list and got 2 +1s: https://mail-archives.apache.org/mod_mbox/incubator-impala-dev/201612.mbox/browser . Here is what I proposed:
One problem we've seen a lot is that certain config values, e.g. IMPALA_TOOLCHAIN_BUILD_ID can be overridden by preexisting environment variables. This is useful for testing against alternate components but this makes the config variables "sticky" and leads to confusing errors if you re-source a changed impala-config.sh and get a mix of old and new config values. E.g. I've seen multiple people run into confusing errors where it looks like files are missing from the toolchain s3 bucket.
My idea is that we should remove this overriding mechanism and add an alternative one without the problem based on additional config files. impala-config.sh would determine the default values, which could be overridden by additional config values per-branch or in the local dev environment.
My initial idea is to have:
./bin/impala-config.sh ./bin/impala-config-branch.sh ./bin/impala-config-local.sh
impala-config-branch.sh would be blank by default and version-controlled, and could be used on release/development branches to override particular config values. This would make it simpler to merge and rebase branches.
impala-config-local.sh would be non-existent by default and added to .gitignore. Users can then put whatever values they want for local testing.
Sourcing impala-config.sh would cause the config to be fully reset, avoiding any staleness issues.