Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4653

Revamp impala-config.sh to avoid annoying "sticky config variables" problem

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Infrastructure
    • Labels:
      None

      Description

      I brought up this idea on the mailing list and got 2 +1s: https://mail-archives.apache.org/mod_mbox/incubator-impala-dev/201612.mbox/browser . Here is what I proposed:

      One problem we've seen a lot is that certain config values, e.g. IMPALA_TOOLCHAIN_BUILD_ID can be overridden by preexisting environment variables. This is useful for testing against alternate components but this makes the config variables "sticky" and leads to confusing errors if you re-source a changed impala-config.sh and get a mix of old and new config values. E.g. I've seen multiple people run into confusing errors where it looks like files are missing from the toolchain s3 bucket.

      My idea is that we should remove this overriding mechanism and add an alternative one without the problem based on additional config files. impala-config.sh would determine the default values, which could be overridden by additional config values per-branch or in the local dev environment.

      My initial idea is to have:

        ./bin/impala-config.sh
        ./bin/impala-config-branch.sh
        ./bin/impala-config-local.sh
      

      impala-config-branch.sh would be blank by default and version-controlled, and could be used on release/development branches to override particular config values. This would make it simpler to merge and rebase branches.

      impala-config-local.sh would be non-existent by default and added to .gitignore. Users can then put whatever values they want for local testing.

      Sourcing impala-config.sh would cause the config to be fully reset, avoiding any staleness issues.

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-4653: fix sticky config variable problem

        Previously we could get a developer's shell into a bad state where a
        value of a config variable from a previous impala-config.sh version
        would override the value from the new impala-config.sh version.

        This change adds a new mechanism to override settings locally by adding
        settings to impala-config-local.sh. This alternative approach is more
        robust, because the config variables will be reset to the intended
        values when impala-config.sh is re-sourced.

        impala-config-branch.sh can also be used to override settings in a
        version-controlled way, e.g. to support having different settings for
        different branches.

        I did not convert all variables to use this approach, since many people
        and Jenkins jobs depend on setting these variables from the environment.
        The remaining "sticky" variables are ones where default values should
        not change frequently, e.g. source directory locations and build
        settings.

        Change-Id: I930e2ca825142428d17a6981c77534ab0c8e3489
        Reviewed-on: http://gerrit.cloudera.org:8080/5545
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-4653 : fix sticky config variable problem Previously we could get a developer's shell into a bad state where a value of a config variable from a previous impala-config.sh version would override the value from the new impala-config.sh version. This change adds a new mechanism to override settings locally by adding settings to impala-config-local.sh. This alternative approach is more robust, because the config variables will be reset to the intended values when impala-config.sh is re-sourced. impala-config-branch.sh can also be used to override settings in a version-controlled way, e.g. to support having different settings for different branches. I did not convert all variables to use this approach, since many people and Jenkins jobs depend on setting these variables from the environment. The remaining "sticky" variables are ones where default values should not change frequently, e.g. source directory locations and build settings. Change-Id: I930e2ca825142428d17a6981c77534ab0c8e3489 Reviewed-on: http://gerrit.cloudera.org:8080/5545 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development